Marine environment real-time monitoring modeling system and method based on multi-modal fusion

Abstract

The invention provides a marine environment real-time monitoring modeling system and method based on multi-modal fusion, and the system comprises a data collection module which is used for collecting marine satellite remote sensing data, buoy sensor data and public crowdsourcing data, and carrying out the preprocessing, and then generating multi-modal data; the multi-modal fusion module is used for registering the multi-modal data in time and space, calculating weights of various data sources in real time, and fusing the registered multi-modal data based on the weights to obtain fused marine environment data; and the real-time three-dimensional modeling module is used for constructing a real-time three-dimensional model of the marine environment by utilizing a three-dimensional modeling technology based on the fused marine environment data, and dynamically updating the real-time three-dimensional model of the marine environment according to the new multi-modal data. By integrating satellite remote sensing, buoy sensor and public crowdsourcing data and utilizing a dynamic weight distribution algorithm, real-time three-dimensional modeling of air-sky-ground-sea multi-source data is realized, so that the marine environment condition is comprehensively and accurately reflected.

Description

Technical Field

[0001] The present invention relates to the technical field of marine monitoring, and particularly relates to a real-time monitoring and modeling system and method for marine environment based on multi-modal fusion. Background Art

[0002] The marine environment is complex and changeable. Accurate and real-time monitoring of it is of crucial significance for many aspects such as marine resource development, marine ecological protection, and safety of maritime transportation. Traditional marine environment monitoring means mainly include satellite remote sensing monitoring, buoy sensor monitoring, etc. Satellite remote sensing can obtain information of large-area marine regions from a macroscopic scale, such as sea surface temperature, sea color, etc.; buoy sensors can measure parameters such as seawater temperature, salinity, dissolved oxygen, etc. in real time at specific positions. However, a single monitoring means has limitations. The resolution of satellite remote sensing data is limited, and it is difficult to accurately capture local small-scale marine environment changes; the monitoring range of buoy sensors is limited by their deployment positions and cannot comprehensively cover the vast marine regions. In addition, with the improvement of the public's awareness of marine protection, public crowdsourcing data (such as marine observation information provided by diving enthusiasts, fishermen, etc.) contains rich local marine environment details but has not been fully and effectively utilized. Therefore, how to integrate multi-source data to achieve comprehensive, real-time, and accurate monitoring and modeling of the marine environment has become an urgent problem to be solved. Summary of the Invention

[0003] The present invention provides a real-time monitoring and modeling system and method for marine environment based on multi-modal fusion. By integrating satellite remote sensing, buoy sensors and public crowdsourcing data, and using a dynamic weight allocation algorithm, real-time three-dimensional modeling of multi-source data of air-space-earth-sea is realized, so as to comprehensively and accurately reflect the marine environment status.

[0004] The present invention provides a real-time monitoring and modeling system for marine environment based on multi-modal fusion, including: A data acquisition module, configured to collect marine satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generate multi-modal data after preprocessing; A multi-modal fusion module, configured to register multi-modal data in terms of time and space, and calculate the weights of various data sources in real time, and fuse the registered multi-modal data based on the weights to obtain fused marine environment data; A real-time three-dimensional modeling module, configured to construct a real-time three-dimensional model of the marine environment by using three-dimensional modeling technology based on the fused marine environment data, and dynamically update the real-time three-dimensional model of the marine environment according to new multi-modal data.

[0005] Preferably, in a real-time monitoring and modeling system for marine environment based on multi-modal fusion, the data acquisition module includes: The satellite remote sensing data acquisition unit is used to receive the ocean remote sensing image data sent by the satellite, send it to the data preprocessing unit, and parse the preprocessed ocean remote sensing image data to generate ocean satellite remote sensing data; The buoy sensor data acquisition unit is used to establish a communication connection with the buoy sensors distributed everywhere in the ocean, and collect in real time the seawater data measured by various sensors on the buoy to generate buoy sensor data; The public crowdsourcing data acquisition unit is used to collect ocean-related data on the Internet of Things based on big data technology to obtain public crowdsourcing data; The data preprocessing unit is used to process the ocean remote sensing image data, buoy sensor data, and public crowdsourcing data respectively based on the corresponding preprocessing rules to generate multi-modal data.

[0006] Preferably, in a real-time monitoring and modeling system for ocean environment based on multi-modal fusion, the data preprocessing unit includes: The satellite remote sensing data preprocessing sub-unit is used to perform radiometric correction, atmospheric correction, and geometric correction on the ocean remote sensing image to obtain the preprocessed ocean remote sensing image data; The buoy sensor data preprocessing sub-unit is used to denoise and filter the seawater data collected by the buoy sensor; The public crowdsourcing data preprocessing sub-unit is used to verify the authenticity and classify and sort the ocean-related data uploaded by the public on the Internet of Things to obtain public crowdsourcing data. Preferably, in a real-time monitoring and modeling system for ocean environment based on multi-modal fusion, the public crowdsourcing data preprocessing sub-unit includes: The key information judgment sub-unit is used to obtain the release time, release location, and uploaded text information of the ocean-related data uploaded by the public; Perform semantic recognition and keyword extraction on the uploaded text information, and judge whether the uploaded text contains the ocean geographical location information of the ocean-related data uploaded by the work according to the semantic result and keyword extraction result; The first data verification sub-unit is used to, if the uploaded text contains the ocean geographical location information of the ocean-related data uploaded by the work, predict the actual observation time interval of the ocean-related data based on the distance difference between the ocean geographical location information and the released location, in combination with the release time; Based on the actual observation time interval and the ocean geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics; Correspondingly, combine the semantic recognition result and the text keywords to determine the main phenomenon described by the user; According to the main phenomena described by the user, extract the key features of the phenomena from the publicly uploaded ocean-related data. Based on comparing the key features of the phenomena with multiple sea surface change features respectively, when the similarity between any sea surface change feature and the key feature of the phenomenon is greater than or equal to the preset value, it is determined that the publicly uploaded ocean-related data is real data; The second data verification subunit is used to, if the uploaded text does not contain the ocean geographical location information of the ocean-related data uploaded for work, extract the marine geological features of the publicly uploaded ocean-related data, conduct relevant ocean retrievals based on the marine geological features, and determine whether there is a suspected target ocean in the preset ocean information database; If not, it is determined that the publicly uploaded ocean-related data is fake data; If it exists, obtain the key features of the phenomenon and the corresponding regional features of the phenomenon occurrence area. Based on the local environmental data corresponding to the geographical location of the suspected target ocean, combined with the characteristics of marine biological activities and marine activities, speculate whether the phenomenon described by the publicly uploaded ocean-related data may exist; If so, obtain the geographical location information corresponding to the suspected ocean and send it to the first data verification subunit for secondary determination; Otherwise, it is determined that the publicly uploaded ocean-related data is fake data.

[0007] Preferably, in a real-time monitoring and modeling system for ocean environment based on multi-modal fusion, the multi-modal fusion module includes: The data registration unit is used to register multi-modal data in terms of time and space based on geographic information system technology, determine the corresponding relationship between time and position coordinates among different multi-modal data, and obtain registered multi-modal data; The dynamic weight allocation unit is used to establish a multi-factor decision-making model based on the analytic hierarchy process and the fuzzy comprehensive evaluation method, and obtain the actual needs of the user; Analyze the actual needs to obtain the user's focus on ocean monitoring, and based on the user's focus on ocean monitoring, determine the user's preferred ocean factors. Based on the user's preferred ocean factors, use the multi-factor decision-making model to adjust the data weights of different multi-modal data, and obtain the current weight allocation result of the multi-modal data; The multi-modal data fusion unit is used to fuse the registered multi-modal data according to the current weight allocation result. Preferably, in a real-time monitoring and modeling system for ocean environment based on multi-modal fusion, the real-time three-dimensional modeling module includes: The three-dimensional model construction unit is used to construct a real-time three-dimensional model of the ocean environment based on the fused multi-modal data using three-dimensional modeling technology; A model update and visualization unit for receiving new multi-source data in real time, recalculating weights according to a dynamic weight allocation algorithm, and performing data fusion.

[0008] Preferably, in a real-time monitoring and modeling system for marine environment based on multimodal fusion, a three-dimensional model construction unit includes: A model layering subunit for dividing a marine three-dimensional model into multiple layers based on a marine standard model, and obtaining marine characterization data corresponding to each layer, as well as the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer; A model construction subunit for classifying the obtained fused multimodal data based on the marine characterization data corresponding to each layer, and obtaining multimodal data sets respectively determined for each layer; Adjust the marine standard model based on the multimodal data set, determine the marine three-dimensional model of each layer corresponding to the current ocean, and obtain a primary marine model; At the same time, based on the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer, and the marine three-dimensional model of each layer corresponding to the current ocean, determine the decomposition characteristics of each layer of the current ocean; Fuse each layer corresponding to the primary marine model based on the decomposition characteristics of each layer of the current ocean to obtain a real-time three-dimensional model of the marine environment corresponding to the current one.

[0009] The present invention provides a real-time monitoring and modeling method for marine environment based on multimodal fusion, including: Step 1: Collect marine satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generate multimodal data after preprocessing; Step 2: Register the multimodal data in terms of time and space, and calculate the weights of various data sources in real time, and fuse the registered multimodal data based on the weights to obtain fused marine environment data; Step 3: Based on the fused marine environment data, use three-dimensional modeling technology to construct a real-time three-dimensional model of the marine environment, and dynamically update the real-time three-dimensional model of the marine environment according to new multimodal data.

[0010] Preferably, in a real-time monitoring and modeling method for marine environment based on multimodal fusion, verifying the authenticity of public crowdsourcing data includes: Step A: Obtain the release time, release location, and upload copy information of the marine-related data uploaded publicly; Perform semantic recognition and keyword extraction on the upload copy information, and judge whether the upload copy contains the marine geographical location information of the marine-related data uploaded for work according to the semantic result and the keyword extraction result; Step B: If the uploaded text contains the ocean geographical location information of the ocean-related data uploaded for work, then based on the distance difference between the published locations of the ocean geographical location information, combined with the publication time, predict the actual observation time interval of the ocean-related data; Based on the actual observation time interval and the ocean geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics; Correspondingly, combine the semantic recognition result and the text keywords to determine the main described phenomenon of the user; According to the main described phenomenon of the user, extract the key features of the phenomenon from the publicly uploaded ocean-related data. Based on comparing the key features of the phenomenon with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded ocean-related data is real data; Step C: If the uploaded text does not contain the ocean geographical location information of the ocean-related data uploaded for work, then extract the ocean geological characteristics from the publicly uploaded ocean-related data, conduct relevant ocean retrievals based on the ocean geological characteristics, and determine whether there is a suspected target ocean in the preset ocean information database; If not, determine that the publicly uploaded ocean-related data is false data; If so, obtain the key features of the phenomenon and the corresponding regional characteristics of the phenomenon occurrence. Based on the local environmental data of the geographical location corresponding to the suspected target ocean, combined with the characteristics of marine biological activities and marine activities, speculate whether the phenomenon described by the publicly uploaded ocean-related data may exist; If so, obtain the geographical location information corresponding to the suspected ocean and send it to Step B for secondary determination; Otherwise, determine that the publicly uploaded ocean-related data is false data.

[0011] Preferably, in a real-time monitoring and modeling method for the ocean environment based on multimodal fusion, Step 3 includes: Based on the ocean standard model, divide the ocean three-dimensional model into multiple layers, and obtain the ocean characterization data corresponding to each layer, as well as the characteristics of the changes in marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer; Based on the ocean characterization data corresponding to each layer, classify the obtained fused multimodal data to obtain the multimodal data sets corresponding to each layer respectively; Based on the multimodal data sets, adjust the ocean standard model to determine the ocean three-dimensional model of each layer corresponding to the current ocean, and obtain the primary ocean model; At the same time, based on the characteristics of the changes in marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer, and the ocean three-dimensional model of each layer corresponding to the current ocean, determine the decomposition characteristics of each layer of the current ocean; Fuse each layer corresponding to the primary ocean model based on the decomposition characteristics of each layer of the current ocean to obtain a real-time three-dimensional model of the current corresponding ocean environment.

[0012] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention integrates satellite remote sensing, buoy sensors and public crowdsourcing data, collects information from multiple dimensions of air, space, land and sea, fills the blind spots of traditional monitoring means, and the diversified data collection method enables the system to quickly capture the dynamic changes of the ocean environment. Then, through the registration of multi-modal data in time and space, it is ensured that data from different sources can be compared and analyzed in the same spatio-temporal framework. The dynamic weight allocation algorithm can dynamically adjust the model according to different monitoring indicators according to the needs of ocean research. For example, when monitoring the ocean surface temperature, a higher weight is given to satellite remote sensing data; when monitoring the local marine biodiversity, the weight of public crowdsourcing data is increased. Based on the weights, the registered data is fused, giving full play to the advantages of each data source, effectively making up for the limitations of a single data source, significantly improving the accuracy of ocean environment monitoring, making the monitoring results more in line with the actual ocean environment conditions. Finally, a real-time three-dimensional model is constructed based on the fused ocean environment data, converting the abstract ocean environment data into an intuitive three-dimensional visualization scene. The model covers multiple layers such as the ocean surface, the interior of the water body and the seabed topography, vividly showing the three-dimensional spatial distribution characteristics of the ocean environment, enabling a quick grasp of the overall picture of the ocean environment, providing an intuitive and efficient information display method for ocean scientific research and management decision-making, and receiving new multi-modal data in real time, recalculating weights and fusing data according to the dynamic weight allocation algorithm, and updating the three-dimensional model in time, so that the model can closely track the real-time changes of the ocean environment, providing a powerful tool for the dynamic monitoring and scientific management of the ocean environment.

[0013] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained by the structures specifically pointed out in this application document.

[0014] The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments. Description of the Drawings

[0015] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention. In the drawings: Figure 1 It is a structural diagram of a real-time monitoring and modeling system for ocean environment based on multi-modal fusion in an embodiment of the present invention; Figure 2This is the structural diagram of the data acquisition module of a real-time marine environment monitoring and modeling system based on multimodal fusion in an embodiment of the present invention; Figure 3 This is the structural diagram of the multimodal fusion module of a real-time marine environment monitoring and modeling system based on multimodal fusion in an embodiment of the present invention; Figure 4 This is the structural diagram of the real-time 3D modeling module of a real-time marine environment monitoring and modeling system based on multimodal fusion in an embodiment of the present invention; Figure 5 This is the flowchart of a real-time marine environment monitoring and modeling method based on multimodal fusion in an embodiment of the present invention. Detailed implementation manners

[0016] The following describes the preferred embodiments of the present invention with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0017] Embodiment 1: The present invention provides a real-time marine environment monitoring and modeling system based on multimodal fusion, as Figure 1 shown, including: A data acquisition module, configured to collect marine satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generate multimodal data after preprocessing; A multimodal fusion module, configured to register multimodal data in terms of time and space, and calculate the weights of various data sources in real time, and fuse the registered multimodal data based on the weights to obtain fused marine environment data; A real-time 3D modeling module, configured to construct a real-time 3D model of the marine environment based on the fused marine environment data by using 3D modeling technology, and dynamically update the real-time 3D model of the marine environment according to new multimodal data.

[0018] Beneficial effects of the above technical solution: The present invention integrates satellite remote sensing, buoy sensors, and public crowdsourcing data, collects information from multiple dimensions of air, space, land, and sea, fills the blind spots of traditional monitoring means, and the diversified data collection method enables the system to quickly capture the dynamic changes of the marine environment. Then, through the registration of multi-modal data in time and space, it ensures that data from different sources can be compared and analyzed in the same spatio-temporal framework. The dynamic weight allocation algorithm can dynamically adjust the model according to different monitoring indicators based on the needs of marine research. For example, when monitoring the sea surface temperature, a higher weight is given to satellite remote sensing data; when monitoring the local marine biodiversity, the weight of public crowdsourcing data is increased. Based on the weights, the registered data is fused, giving full play to the advantages of each data source, effectively making up for the limitations of a single data source, significantly improving the accuracy of marine environment monitoring, making the monitoring results more in line with the actual marine environment conditions. Finally, a real-time three-dimensional model is constructed based on the fused marine environment data, converting the abstract marine environment data into an intuitive three-dimensional visualization scene. The model covers multiple levels such as the sea surface, the interior of the water body, and the seabed topography, vividly displaying the three-dimensional spatial distribution characteristics of the marine environment, enabling a quick grasp of the overall picture of the marine environment, providing an intuitive and efficient information display method for marine scientific research and management decision-making, and receiving new multi-modal data in real time, recalculating weights and fusing data according to the dynamic weight allocation algorithm, and updating the three-dimensional model in a timely manner, enabling the model to closely track the real-time changes of the marine environment and providing a powerful tool for the dynamic monitoring and scientific management of the marine environment.

[0019] Embodiment 2: On the basis of Embodiment 1, the data acquisition module, as Figure 2 shown, includes: A satellite remote sensing data acquisition unit, which is used to receive the marine remote sensing image data sent by the satellite and parse the marine remote sensing image data to generate marine satellite remote sensing data; A buoy sensor data acquisition unit, which is used to establish a communication connection with buoy sensors distributed in various parts of the ocean, and collect in real time the seawater data measured by various sensors on the buoy to generate buoy sensor data; A public crowdsourcing data acquisition unit, which is used to collect ocean-related data on the Internet of Things based on big data technology to obtain public crowdsourcing data; A data preprocessing unit, which is used to process the marine satellite remote sensing data, buoy sensor data, and public crowdsourcing data respectively based on the corresponding preprocessing rules to generate multi-modal data.

[0020] In this embodiment, the marine satellite remote sensing data includes sea surface temperature, sea color, sea surface height, and sea surface biological activity information.

[0021] In this embodiment, the seawater data includes temperature, salinity, dissolved oxygen, flow velocity, flow direction, etc.

[0022] In this embodiment, the ocean-related data includes photos of marine organisms taken, descriptions of seawater color changes, records of local ocean anomalies, etc.

[0023] Advantages of the above technical solution: The present invention collects information from multiple dimensions of space, sky, land, and sea, filling the blind spots of traditional monitoring means. The diversified data collection method can comprehensively describe the ocean environment from multiple dimensions, make up for the limitations of a single data source, and achieve more accurate and comprehensive monitoring and modeling of the ocean environment.

[0024] Embodiment 3: Based on Embodiment 2, the data preprocessing unit includes: A satellite remote sensing data preprocessing subunit, which is used to perform radiometric correction, atmospheric correction, and geometric correction processing on ocean remote sensing images to obtain preprocessed ocean remote sensing image data; A buoy sensor data preprocessing subunit, which is used to perform denoising and filtering processing on the data collected by the buoy sensor; A public crowdsourcing data preprocessing subunit, which is used to perform authenticity verification and classification on the ocean-related data uploaded by the public on the Internet of Things to obtain public crowdsourcing data Advantages of the above technical solution: The present invention performs radiometric correction, atmospheric correction, and geometric correction processing on satellite remote sensing images to eliminate errors caused by factors such as sensor characteristics and atmospheric interference, and improve the accuracy and comparability of image data. At the same time, perform denoising and filtering processing on the data collected by the buoy sensor to remove abnormal data caused by sensor noise, electromagnetic interference, etc. For example, use the Kalman filter algorithm to denoise time series data such as seawater temperature and salinity. And perform authenticity verification and classification on the ocean-related data uploaded by the public to ensure the accuracy and availability of public crowdsourcing data.

[0025] Embodiment 4: Based on Embodiment 3, the public crowdsourcing data preprocessing subunit includes: A key information judgment subunit, which is used to obtain the release time, release location, and upload copy information of the ocean-related data uploaded publicly; Perform semantic recognition and keyword extraction on the upload copy information, and judge whether the upload copy contains the ocean geographical location information of the ocean-related data uploaded publicly according to the semantic result and keyword extraction result; A first data verification subunit, which is used to predict the actual observation time interval of the ocean-related data based on the distance difference between the ocean geographical location information and the released location and in combination with the release time if the upload copy contains the ocean geographical location information of the ocean-related data uploaded publicly; Based on the actual observation time interval and ocean geographical location information, obtain corresponding multiple target remote sensing images and their sea surface change characteristics; Correspondingly, combine the semantic recognition results and copywriting keywords to determine the main described phenomena of the user; According to the main described phenomena of the user, extract the key characteristics of the phenomena from the publicly uploaded ocean-related data. Based on comparing the key characteristics of the phenomena with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key characteristic of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded ocean-related data is real data; The second data verification subunit is used to, if the uploaded copywriting does not contain the ocean geographical location information of the publicly uploaded ocean-related data, extract the ocean geological characteristics of the publicly uploaded ocean-related data, conduct relevant ocean retrievals based on the ocean geological characteristics, and determine whether there is a suspected target ocean in the preset ocean information database; If not, determine that the publicly uploaded ocean-related data is false data; If so, obtain the key characteristics of the phenomenon and the corresponding characteristics of the phenomenon occurrence area, and based on the local environmental data corresponding to the geographical location of the suspected target ocean, combine the characteristics of marine biological activities and marine activities to infer whether the phenomenon described in the publicly uploaded ocean-related data may exist; If so, obtain the geographical location information corresponding to the suspected ocean and send it to the first data verification subunit for determination; Otherwise, determine that the publicly uploaded ocean-related data is false data.

[0026] Beneficial effects of the above technical solution: The present invention conducts authenticity verification and classification and sorting on the publicly uploaded ocean-related data, ensuring the accuracy and availability of the public crowdsourcing data.

[0027] Embodiment 5: Based on Embodiment 1, the multimodal fusion module, as Figure 3 shown, includes: The data registration unit is used to register multimodal data in terms of time and space based on geographic information system technology, determine the corresponding relationship between time and position coordinates among different multimodal data, and obtain registered multimodal data; The dynamic weight allocation unit is used to establish a multi-factor decision-making model based on the analytic hierarchy process and the fuzzy comprehensive evaluation method to obtain the actual needs of the user; Analyze the actual needs to obtain the user's focus on ocean monitoring, and based on the user's focus on ocean monitoring, determine the user's preferred ocean factors. Based on the user's preferred ocean factors, use the multi-factor decision-making model to adjust the data weights of different multimodal data to obtain the current weight allocation result of the multimodal data; The multimodal data fusion unit is used to fuse the registered multimodal data according to the current weight allocation result. In this embodiment, for numerical data (such as seawater temperature, salinity, etc.), the weighted average method is used for fusion; for image data (such as satellite remote sensing images and ocean photos taken by the public), a feature-based fusion method is adopted to extract key features (such as color features, texture features, etc.) in the images for fusion processing, generating the fused ocean environment data.

[0028] Beneficial effects of the above technical solution: According to the present invention, the data registration unit registers multimodal data in terms of time and space based on geographic information system technology, clarifies the corresponding relationship between time and position coordinates among different multimodal data, and generates registered multimodal data, enabling data from various sources such as satellite remote sensing, buoy sensors, and public crowdsourcing to be analyzed within a unified spatio-temporal framework, eliminating data biases caused by spatio-temporal inconsistencies. For example, accurately correlating the large-area ocean surface temperature data obtained by a satellite at a specific time with the seawater temperature data measured by a buoy sensor at a local position at the same moment provides a reliable basis for subsequent fusion and analysis. Subsequently, the dynamic weight allocation unit constructs a multi-factor decision-making model using the analytic hierarchy process and fuzzy comprehensive evaluation method, and by analyzing the actual needs of users, clarifies the focus of users' ocean monitoring, and then determines the ocean factors preferred by users, enabling flexible adjustment of the data proportion of different data sources according to the needs of different users. For example, scientific researchers may be more concerned about biodiversity in the marine ecosystem. Through model adjustment, the weight of the part of public crowdsourcing data regarding marine biological observations will increase; while the maritime department focuses more on ocean meteorology and ocean current data when monitoring maritime traffic safety, and the model will correspondingly increase the weights of data such as sea surface height, wind speed and direction obtained by satellite remote sensing and the flow velocity and direction measured by buoy sensors, so that the monitoring results output by the system are more in line with the actual needs of users. Then, according to the ocean factors preferred by users, the multi-factor decision-making model is used to adjust the data weights of different multimodal data to obtain the current weight allocation result. For example, when monitoring the ocean surface temperature, the weight of satellite remote sensing data is set to 0.6, the weight of buoy sensor data is set to 0.3, and the weight of public crowdsourcing data is set to 0.1; when monitoring marine biodiversity, the weight of satellite remote sensing data is set to 0.2, the weight of buoy sensor data is set to 0.2, and the weight of public crowdsourcing data is set to 0.6. Finally, the multimodal data fusion unit fuses the registered multimodal data based on this result, ensuring that in the data fusion process, more important and user-demand-compliant data can be fully utilized, avoiding fusion result biases caused by unreasonable data weights, and enhancing the accuracy and reliability of ocean environment monitoring results.

[0029] Embodiment 6: On the basis of Embodiment 1, a real-time three-dimensional modeling module, such asFigure 4 As shown in the figure, it includes: A three-dimensional model construction unit, which is used to construct a real-time three-dimensional model of the marine environment based on the fused marine environment data by using three-dimensional modeling technology; A model update and visualization unit, which is used to receive new multi-source data in real time, recalculate weights according to the dynamic weight allocation algorithm, and perform data fusion.

[0030] Beneficial effects of the above technical solution: Based on the fused marine environment data, a real-time three-dimensional model is constructed, converting the abstract marine environment data into an intuitive three-dimensional visualization scene. The model covers multiple levels such as the ocean surface, the interior of the water body, and the seabed topography, vividly showing the three-dimensional spatial distribution characteristics of the marine environment, enabling a quick grasp of the overall picture of the marine environment, providing an intuitive and efficient information display method for marine scientific research and management decision-making, and receiving new multi-modal data in real time, recalculating weights and fusing data according to the dynamic weight allocation algorithm, and updating the three-dimensional model in a timely manner, enabling the model to closely track the real-time changes of the marine environment, and providing a powerful tool for the dynamic monitoring and scientific management of the marine environment.

[0031] Embodiment 7: Based on Embodiment 6, the three-dimensional model construction unit includes: A model layering sub-unit, which is used to divide the marine three-dimensional model into multiple levels based on the marine standard model, and obtain the marine characterization data corresponding to each level, as well as the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each level; A model construction sub-unit, which is used to classify the fused multi-modal data based on the marine characterization data corresponding to each level, and obtain the multi-modal data sets corresponding to each level respectively; Adjust the marine standard model based on the multi-modal data set, determine the marine three-dimensional models of each level corresponding to the current ocean, and obtain a primary marine model; At the same time, based on the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each level, and the marine three-dimensional models of each level corresponding to the current ocean, determine the decomposition characteristics of each level of the current ocean; Fuse the corresponding levels of the primary marine model based on the decomposition characteristics of each level of the current ocean to obtain the real-time three-dimensional model of the current corresponding marine environment.

[0032] Beneficial effects of the above technical solution: The present invention is based on the marine standard model to establish each level of the current ocean respectively and then fuse multiple levels, effectively accelerating the establishment speed of the real-time three-dimensional model of the marine environment and realizing the rapid establishment of the real-time three-dimensional model of the marine environment.

[0033] Embodiment 8: The present invention provides a real-time monitoring and modeling method for marine environment based on multimodal fusion, as Figure 5 shown, including: Step 1: Collect marine satellite remote sensing data, buoy sensor data and public crowdsourcing data. After preprocessing, generate multimodal data; Step 2: Register the multimodal data in terms of time and space, and calculate the weights of various data sources in real time. Based on the weights, fuse the registered multimodal data to obtain fused marine environment data; Step 3: Based on the fused marine environment data, use three-dimensional modeling technology to construct a real-time three-dimensional model of the marine environment, and dynamically update the real-time three-dimensional model of the marine environment according to the new multimodal data.

[0034] Beneficial effects of the above technical solution: The present invention integrates satellite remote sensing, buoy sensors and public crowdsourcing data, collects information from multiple dimensions of air, space, land and sea, fills the blind spots of traditional monitoring means, and the diversified data collection method enables the system to quickly capture the dynamic changes of the marine environment. Then, by registering the multimodal data in terms of time and space, it ensures that data from different sources can be compared and analyzed in the same spatio-temporal framework. The dynamic weight allocation algorithm can dynamically adjust the model according to different monitoring indicators and the needs of marine research. For example, when monitoring the sea surface temperature, a higher weight is given to satellite remote sensing data; when monitoring the local marine biodiversity, the weight of public crowdsourcing data is increased. Based on the weights, the registered data are fused, giving full play to the advantages of each data source, effectively making up for the limitations of a single data source, significantly improving the accuracy of marine environment monitoring, making the monitoring results more in line with the actual marine environment conditions. Finally, based on the fused marine environment data, a real-time three-dimensional model is constructed, transforming the abstract marine environment data into an intuitive three-dimensional visualization scene. The model covers multiple levels such as the sea surface, the water body interior and the seabed topography, vividly showing the three-dimensional spatial distribution characteristics of the marine environment, enabling a quick grasp of the overall picture of the marine environment, providing an intuitive and efficient information display method for marine scientific research and management decision-making, and receiving new multimodal data in real time, recalculating the weights and fusing the data according to the dynamic weight allocation algorithm, and updating the three-dimensional model in a timely manner, enabling the model to closely track the real-time changes of the marine environment, and providing a powerful tool for the dynamic monitoring and scientific management of the marine environment.

[0035] Example 9: Based on Example 8, a real-time monitoring and modeling method for marine environment based on multimodal fusion includes verifying the authenticity of public crowdsourcing data, including: Step A: Obtain the release time, release location and uploaded text information of the marine-related data uploaded publicly; Perform semantic recognition and keyword extraction on the uploaded text information. Based on the semantic results and keyword extraction results, determine whether the uploaded text contains the marine geographical location information of the marine-related data uploaded for work; Step B: If the uploaded text contains the marine geographical location information of the marine-related data uploaded for work, then based on the distance difference between the marine geographical location information and the published location, combined with the publication time, predict the actual observation time interval of the marine-related data; Based on the actual observation time interval and the marine geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics; Correspondingly, combine the semantic recognition results and the text keywords to determine the main described phenomenon of the user; According to the main described phenomenon of the user, extract the key features of the phenomenon from the publicly uploaded marine-related data. Based on comparing the key features of the phenomenon with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded marine-related data is real data; Step C: If the uploaded text does not contain the marine geographical location information of the marine-related data uploaded for work, then extract the marine geological features from the publicly uploaded marine-related data, and perform relevant marine retrievals based on the marine geological features to determine whether there is a suspected target ocean in the preset marine information database; If not, determine that the publicly uploaded marine-related data is fake data; If so, obtain the key features of the phenomenon and the corresponding regional features of the phenomenon occurrence. Based on the local environmental data corresponding to the geographical location of the suspected target ocean, combined with the marine biological activity characteristics and marine activity characteristics, speculate whether the phenomenon described by the publicly uploaded marine-related data may exist; If so, obtain the geographical location information corresponding to the suspected ocean and send it to Step B for secondary determination; Otherwise, determine that the publicly uploaded marine-related data is fake data.

[0036] Beneficial effects of the above technical solution: By obtaining the release time, location, and uploaded text information of the data uploaded by the public, the present invention performs semantic recognition and keyword extraction on the text, determines whether it contains marine geographical location information, quickly screens out the data containing key geographical information, and clarifies the direction for subsequent more in-depth verification work. For example, among the data uploaded by the public, accurately identify those records that clearly mention the observation location, concentrate the limited verification resources on these potentially valid data, improve the data screening efficiency, and avoid unnecessary subsequent processing of a large amount of irrelevant or invalid data; and when the uploaded text contains marine geographical location information, based on the distance difference between the location information and the release location and the release time, predict the actual observation time interval, which helps to combine the data uploaded by the public with information such as satellite remote sensing images at the corresponding time and location. For example, by reasonably inferring the observation time, multiple target remote sensing images of the corresponding area and their sea surface change characteristics during this time period can be accurately obtained. At the same time, combined with text analysis, determine the main phenomena described by the user, and extract the key characteristics of the phenomena for comparison with the sea surface change characteristics. Through the method of multi-information fusion and comparison, the authenticity of the data can be verified from multiple dimensions, greatly improving the accuracy of data verification and reducing misjudgments caused by data mismatch or false descriptions; when the uploaded text does not contain marine geographical location information, extract the marine geological characteristics of the data, and retrieve the suspected target ocean in the preset marine information database. If there is a suspected target ocean, combine the local environmental data, marine organisms, and activity characteristics to speculate whether the described phenomena may exist, and it is possible to discover data that may be related to real marine phenomena although the geographical location is not clearly stated through other characteristics. For example, some data describing the behavior of unique marine organisms or abnormal marine phenomena, even if not marked with a location, may discover potential real observation records through geological feature matching and phenomenon rationality speculation. If a suspected target ocean exists, send it to step B for secondary determination to further ensure the authenticity of the data, avoid missing valuable data, and the authenticity determination mechanism for true and false data can effectively eliminate false, incorrect, or misleading data uploaded by the public, ensure that the public crowdsourcing data entering the marine environmental monitoring system has high quality, ensure the accuracy and availability of the public crowdsourcing data, and exclude false data to avoid its interference with subsequent data analysis, model construction, and decision-making, and improve the reliability and scientificity of the entire marine environmental monitoring system.

[0037] Embodiment 10: Based on the embodiment 8, step 3 includes: Divide the three-dimensional ocean model into multiple layers based on the ocean standard model, and obtain the corresponding ocean characterization data for each layer, as well as the change characteristics of marine organism species and the seawater decomposition data characteristics at the boundaries of each layer; Classify the obtained fused multi-modal data based on the ocean representation data corresponding to each layer, and obtain the multi-modal data sets corresponding to each layer respectively. Adjust the ocean standard model based on the multi-modal data set, determine the three-dimensional ocean models of each layer corresponding to the current ocean, and obtain the primary ocean model. At the same time, based on the characteristics of the changes in marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer, determine the decomposition characteristics of each layer of the current ocean for the three-dimensional ocean models of each layer corresponding to the current ocean. Fuse each layer corresponding to the primary ocean model based on the decomposition characteristics of each layer of the current ocean to obtain the real-time three-dimensional model of the current corresponding ocean environment.

[0038] Advantages of the above technical solution: Based on the ocean standard model, this invention establishes each layer of the current ocean separately and then fuses multiple layers, effectively accelerating the establishment speed of the real-time three-dimensional model of the ocean environment and realizing the rapid establishment of the real-time three-dimensional model of the ocean environment. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A real-time monitoring modeling system for marine environment based on multimodal fusion, characterized in that: include: The data acquisition module is used to collect ocean satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generate multimodal data after preprocessing; A multimodal fusion module is used to register the multimodal data in time and space, calculate the weights of various data sources in real time, and fuse the registered multimodal data based on the weights to obtain fused marine environment data; The real-time 3D modeling module is used to build a real-time 3D model of the ocean environment based on the fusion of ocean environment data and the use of 3D modeling technology, and dynamically update the real-time 3D model of the ocean environment according to new multimodal data.

2. According to claim 1, a real-time monitoring modeling system for marine environment based on multimodal fusion is characterized in that: Data acquisition module, including: The satellite remote sensing data acquisition unit is used to receive the ocean remote sensing image data sent by the satellite and send it to the data preprocessing unit, and analyze the preprocessed ocean remote sensing image data to generate ocean satellite remote sensing data; The buoy sensor data acquisition unit is used to establish communication connections with buoy sensors distributed throughout the ocean, collect seawater data measured by various sensors on the buoy in real time, and generate buoy sensor data; A public crowdsourcing data collection unit, used to collect ocean-related data on the Internet of Things based on big data technology to obtain public crowdsourcing data; The data preprocessing unit is used to process the ocean remote sensing image data, buoy sensor data and public crowdsourcing data respectively based on corresponding preprocessing rules to generate multimodal data.

3. The marine environment real-time monitoring modeling system based on multimodal fusion according to claim 2 is characterized in that: Data preprocessing unit, including: The satellite remote sensing data preprocessing subunit is used to perform radiation correction, atmospheric correction and geometric correction on the ocean remote sensing image to obtain preprocessed ocean remote sensing image data; The buoy sensor data preprocessing subunit is used to perform denoising and filtering on the seawater data collected by the buoy sensor; The public crowdsourcing data preprocessing subunit is used to verify the authenticity and classify the ocean-related data uploaded by the public on the Internet of Things to obtain public crowdsourcing data.

4. The marine environment real-time monitoring modeling system based on multimodal fusion according to claim 3 is characterized in that: The public crowdsourcing data preprocessing subunit includes: The key information judgment subunit is used to obtain the release time, release location and upload copy information of the publicly uploaded ocean-related data; Performing semantic recognition and keyword extraction on the uploaded document information, and judging whether the uploaded document contains the ocean geographic location information of the ocean-related data uploaded by the work according to the semantic results and the keyword extraction results; A first data verification subunit is used to predict the actual observation time interval of the ocean-related data based on the distance difference between the published locations of the ocean-related data and the publishing time if the uploaded document contains the ocean geographic location information of the ocean-related data uploaded by the work; Based on the actual observation time interval and ocean geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics; Correspondingly, the semantic recognition results and copywriting keywords are combined to determine the main phenomenon described by the user; Extract key features of the publicly uploaded ocean-related data according to the phenomenon mainly described by the user, and compare the key features of the phenomenon with multiple sea surface change features respectively. When the similarity between any sea surface change feature and the key features of the phenomenon is greater than or equal to a preset value, the publicly uploaded ocean-related data is determined to be true data; The second data verification subunit is used to extract the marine geological characteristics of the publicly uploaded marine related data if the uploaded document does not contain the marine geographical location information of the marine related data uploaded by the work, and perform relevant marine retrieval based on the marine geological characteristics to determine whether there is a suspected target ocean in the preset marine information database; If it does not exist, the ocean-related data uploaded by the public is judged to be false data; If it exists, the key features of the phenomenon and the corresponding characteristics of the region where the phenomenon occurs are obtained. Based on the local environmental data of the geographical location corresponding to the suspected target ocean, combined with the characteristics of marine biological activities and marine activity characteristics, it is inferred whether the phenomenon described in the publicly uploaded ocean-related data is likely to exist; If so, the geographical location information corresponding to the suspected ocean is obtained and sent to the first data verification subunit for secondary determination; Otherwise, the ocean-related data uploaded by the public is judged to be false data.

5. The marine environment real-time monitoring modeling system based on multimodal fusion according to claim 1 is characterized in that: Multimodal fusion module, including: A data registration unit is used to register the multimodal data in time and space based on the information system technology, determine the corresponding relationship between the time and position coordinates of different multimodal data, and obtain the registered multimodal data; Dynamic weight allocation unit, used to establish a multi-factor decision model based on hierarchical analysis method and fuzzy comprehensive evaluation method to obtain the actual needs of users; Analyze the actual needs to obtain the user's marine monitoring focus, and determine the user's preferred marine factors based on the marine monitoring focus. Based on the user's preferred marine factors, use a multi-factor decision model to adjust the data weights of different multimodal data to obtain the current weight distribution result of the multimodal data; The multimodal data fusion unit is used to fuse the registered multimodal data according to the current weight distribution result.

6. The marine environment real-time monitoring modeling system based on multimodal fusion according to claim 1 is characterized in that: Real-time 3D modeling module, including: A three-dimensional model building unit, used to build a real-time three-dimensional model of the marine environment based on the fused multi-modal data using three-dimensional modeling technology; The model updating and visualization unit is used to receive new multi-source data in real time, recalculate weights according to the dynamic weight allocation algorithm, and perform data fusion.

7. The marine environment real-time monitoring modeling system based on multimodal fusion according to claim 6 is characterized in that: 3D model building unit, including: The model stratification subunit is used to divide the ocean three-dimensional model into multiple levels based on the ocean standard model, and obtain the ocean characterization data corresponding to each level, the characteristics of the change of marine biological species at the boundaries of each level, and the characteristics of seawater decomposition data; The model building subunit is used to classify the acquired fused multimodal data based on the ocean representation data corresponding to each level, and obtain the multimodal data set corresponding to each level respectively; Adjust the ocean standard model based on the multimodal data set, determine the three-dimensional ocean model at each level corresponding to the current ocean, and obtain a primary ocean model; At the same time, based on the changing characteristics of marine biological species at the boundaries of each level and the characteristics of seawater decomposition data, the three-dimensional ocean model of each level corresponding to the current ocean determines the decomposition characteristics of each level of the current ocean; Based on the decomposition characteristics of each layer of the current ocean, the corresponding layers of the primary ocean model are fused to obtain the current corresponding real-time three-dimensional model of the ocean environment.

8. A real-time monitoring modeling method for marine environment based on multimodal fusion, characterized in that: include: Step 1: Collect ocean satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generate multimodal data after preprocessing; Step 2: align the multimodal data in time and space, calculate the weights of various data sources in real time, fuse the aligned multimodal data based on the weights, and obtain fused marine environment data; Step 3: Based on the fused ocean environment data, use 3D modeling technology to build a real-time 3D model of the ocean environment, and dynamically update the real-time 3D model of the ocean environment according to the new multimodal data.

9. The method for real-time monitoring and modeling of marine environment based on multimodal fusion according to claim 8 is characterized in that: Verify the authenticity of public crowdsourced data, including: Step A: Obtain the release time, release location and upload copy information of publicly uploaded ocean-related data; Performing semantic recognition and keyword extraction on the uploaded document information, and judging whether the uploaded document contains the ocean geographic location information of the ocean-related data uploaded by the work according to the semantic results and the keyword extraction results; Step B: If the uploaded document contains the ocean geographic location information of the ocean-related data uploaded by the work, then based on the distance difference between the published locations of the ocean geographic location information and the publishing time, the actual observation time interval of the ocean-related data is predicted; Based on the actual observation time interval and ocean geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics; Correspondingly, the semantic recognition results and copywriting keywords are combined to determine the main phenomenon described by the user; Extract key features of the publicly uploaded ocean-related data according to the phenomenon mainly described by the user, and compare the key features of the phenomenon with multiple sea surface change features respectively. When the similarity between any sea surface change feature and the key features of the phenomenon is greater than or equal to a preset value, the publicly uploaded ocean-related data is determined to be true data; Step C: If the uploaded document does not contain the ocean geographic location information of the ocean-related data uploaded by the work, the ocean geological features of the publicly uploaded ocean-related data are extracted, and the relevant ocean is searched based on the ocean geological features to determine whether there is a suspected target ocean in the preset ocean information database; If it does not exist, the ocean-related data uploaded by the public is judged to be false data; If it exists, the key features of the phenomenon and the corresponding characteristics of the region where the phenomenon occurs are obtained. Based on the local environmental data of the geographical location corresponding to the suspected target ocean, combined with the characteristics of marine biological activities and marine activity characteristics, it is inferred whether the phenomenon described in the publicly uploaded ocean-related data is likely to exist; If so, obtain the geographical location information corresponding to the suspected ocean and send it to step B for secondary determination; Otherwise, the ocean-related data uploaded by the public is judged to be false data.

10. The method for real-time monitoring and modeling of marine environment based on multimodal fusion according to claim 8, characterized in that: Step 3 includes: Divide the ocean three-dimensional model into multiple levels based on the ocean standard model, and obtain the ocean characterization data corresponding to each level, as well as the characteristics of the changes in marine biological species and seawater decomposition data at the boundaries of each level; Based on the ocean representation data corresponding to each level, the fused multimodal data are classified to obtain the multimodal data sets corresponding to each level; Adjust the ocean standard model based on the multimodal data set, determine the three-dimensional ocean model at each level corresponding to the current ocean, and obtain a primary ocean model; At the same time, based on the changing characteristics of marine biological species at the boundaries of each level and the characteristics of seawater decomposition data, the three-dimensional ocean model of each level corresponding to the current ocean determines the decomposition characteristics of each level of the current ocean; Based on the decomposition characteristics of each layer of the current ocean, the corresponding layers of the primary ocean model are fused to obtain the current corresponding real-time three-dimensional model of the ocean environment.

Images