Method and apparatus for visualizing multi-modal environment data, electronic device, and medium

Abstract

This application belongs to the field of computer technology and relates to a method, apparatus, electronic device, and medium for visualizing multimodal environmental data. The method includes: acquiring a multimodal environmental dataset; classifying the multimodal environmental data based on data type to generate categorized environmental data sets; receiving first data association information for the categorized environmental data sets sent by a target terminal, and performing a first association processing on each categorized environmental data based on the first data association information to generate a first associated environmental data set; determining a second data association information set corresponding to the first associated environmental data set based on a relationship prediction algorithm, and merging each first associated environmental data set to obtain a merged environmental data set; further performing data analysis processing to generate an environmental data map, obtaining an environmental data map set; and sending it to the target terminal for visualization. This application improves the analysis efficiency and application value of multimodal environmental data.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to a method, apparatus, electronic device and medium for visualizing multimodal environmental data. Background Technology

[0002] When visualizing multimodal environmental data, how to effectively visualize this data has become an important research topic. Currently, the common methods for visualizing multimodal environmental data are either to directly analyze and visualize the data or to perform hard data association before visualization.

[0003] However, when using the above methods to visualize multimodal environmental data, the following technical problems often arise: Multimodal environmental data often exhibits correlations after combination. However, linking multimodal data solely through hard data association can lead to inconsistencies between the associated environmental data and the actual correlations. This results in the generation of incorrect environmental data maps during environmental data analysis, ultimately rendering the visualization results invalid.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method, apparatus, electronic device, and medium for visualizing multimodal environmental data, thereby overcoming or at least overcoming the shortcomings of the prior art.

[0006] Firstly, this application provides a method for visualizing multimodal environmental data, including: Obtain a multimodal environment dataset, wherein each multimodal environment data in the dataset corresponds to a unique data type; Based on the data type, the multimodal environment data in the multimodal environment dataset is classified to generate a categorized environment data set, where each categorized environment data set corresponds to a data type. The system receives first data association information for a categorized environmental data set sent by the target terminal, and performs first association processing on each categorized environmental data in the categorized environmental data set based on the first data association information to generate a first associated environmental data set. Based on the relationship prediction algorithm, the second data association information set corresponding to the first associated environment data set is determined, and based on the second data association information set, the various first associated environment data sets included in the first associated environment data set are merged to generate at least one merged environment data set, thus obtaining the merged environment data set. Data analysis and processing are performed on the merged environmental data set, and at least one environmental data map is generated based on the data analysis results to obtain an environmental data map set; Each environmental data map in the environmental data map set is sent to the target terminal for visualization.

[0007] Secondly, this application also provides a visualization device for multimodal environmental data, the device comprising: The acquisition unit is used to acquire a multimodal environment dataset, wherein each multimodal environment data in the multimodal environment dataset corresponds to a unique data type; A classification unit is used to classify the multimodal environment data in the multimodal environment dataset based on the data type, and generate a classification environment data set, wherein each classification environment data set corresponds to a data type. The association unit is used to receive first data association information for the classification environment data set sent by the target terminal, and to perform first association processing on each classification environment data in the classification environment data set based on the first data association information to generate the first associated environment data set. The merging unit is used to determine the second data association information set corresponding to the first associated environment data set based on the relationship prediction algorithm, and to merge each of the first associated environment data sets included in the first associated environment data set based on the second data association information set, so as to generate at least one merged environment data set and obtain a merged environment data set. The analysis unit is used to perform data analysis and processing on the merged environmental data set, and to generate at least one environmental data map based on the data analysis results, thereby obtaining an environmental data map set.

[0008] The display unit is used to send each environmental data map in the environmental data map set to the target terminal for visualization.

[0009] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for visualizing multimodal environment data.

[0010] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for visualizing multimodal environment data.

[0011] The above-mentioned at least one technical application used in the embodiments of this application can achieve the following beneficial effects: The multimodal environmental data visualization method provided in this application avoids the generation of erroneous environmental data maps during environmental data analysis, thus preventing invalid visualization results. Specifically, the reason for generating erroneous environmental data maps and thus invalid visualization results during environmental data analysis is that multimodal environmental data often exhibits combined correlations. Simply associating multimodal data through hard data linkages leads to inconsistencies between the associated environmental data and the actual correlations, resulting in erroneous environmental data maps and thus invalid visualization results. Therefore, the multimodal environmental data visualization method provided in this application first obtains a multimodal environmental dataset. This allows for the acquisition of multimodal environmental data. Second, based on the various data types corresponding to the multimodal environmental dataset, the multimodal environmental data included in the dataset are classified to generate categorized environmental data sets. This allows for the classification of environmental data according to data type. Then, the system receives first data association information for the aforementioned categorized environmental data sets from the target terminal, and performs a first association process on each category of environmental data in the aforementioned categorized environmental data sets based on the first data association information to generate a first associated environmental data set. Thus, environmental data can be associated for the first time using the received association information. Next, based on a relationship prediction algorithm, the system determines second data association information corresponding to the aforementioned first associated environmental data set, and merges each first associated environmental data set included in the aforementioned first associated environmental data set based on the second data association information to generate at least one merged environmental data set. Thus, based on the predicted association information, the environmental data after the first association can be associated a second time and merged. Finally, the system performs data analysis processing on the aforementioned merged environmental data set, and generates at least one environmental data map based on the data analysis results to obtain an environmental data map set; each environmental data map in the aforementioned environmental data map set is sent to the aforementioned target terminal for visualization. Thus, an environmental data relationship map can be generated and visualized. Therefore, environmental data can be fully correlated through two associations, thus avoiding the generation of incorrect correlation graphs and invalid visualization results. In summary, this application effectively solves the problems of inconsistencies between correlations and reality, and invalid visualization results caused by traditional hard correlations of multimodal environmental data; it provides users with intuitive and effective data visualization support, significantly improving the analysis efficiency and application value of multimodal environmental data. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a method for visualizing multimodal environment data according to an embodiment of this application is shown; Figure 2 A schematic diagram of the structure of a visualization device for multimodal environment data according to an embodiment of this application is shown; Figure 3 A schematic diagram of the resulting electronic device according to an embodiment of this application is shown. Detailed Implementation

[0013] To make the objectives, technical claims, and advantages of this application clearer, the technical application of this application will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] Figure 1 This illustration shows a flowchart of a method for visualizing multimodal environment data according to an embodiment of this application. Figure 1 As can be seen, this embodiment includes steps S100 to S600: Step S100: Obtain a multimodal environment dataset, wherein each multimodal environment data in the multimodal environment dataset corresponds to a unique data type.

[0015] In some embodiments of this application, the executing entity (e.g., a server) of the multimodal environment data visualization method can acquire a multimodal environment dataset. Each piece of multimodal environment data in the dataset can correspond to a unique data type. This data type can be a data format type, such as video, image, or text.

[0016] Specifically, in some embodiments of this application, a multimodal environment dataset can be obtained through the following steps S110 to S140: Step S110: Receive at least one satellite remote sensing image of the target area transmitted by an associated satellite device, as a multispectral image set. The associated satellite device may be a satellite device connected to the executing entity via a wired or wireless connection. The satellite remote sensing image may be a multispectral image.

[0017] Step S120: Control the target meteorological sensor to collect meteorological data of the target area and generate a meteorological dataset. The meteorological data in the dataset may include, but is not limited to, temperature, humidity, and air pressure. The target meteorological sensor may be a meteorological sensor capable of monitoring the target area. As an example, the meteorological sensor may be a temperature and humidity sensor.

[0018] Step S130: Control the associated unmanned aerial vehicle (UAV) to acquire laser point cloud data and visible light video of the target area, and generate a laser point cloud dataset and visible light video. The aforementioned UAV may be a drone with flight and photography capabilities.

[0019] Step S140: Combine the above-mentioned multispectral image set, the above-mentioned meteorological dataset, the above-mentioned laser point cloud dataset, and the above-mentioned visible light video into a multimodal environment dataset.

[0020] Step S200: Based on the data type, classify the multimodal environment data in the multimodal environment dataset to generate a categorized environment data set, where each categorized environment data set corresponds to a data type.

[0021] In some embodiments of this application, the aforementioned execution entity can classify the various multimodal environment data included in the multimodal environment dataset based on the various data types corresponding to the multimodal environment dataset, to generate a categorized environment data set. Each group in the generated categorized environment data set is denoted as a categorized environment data group, and each categorized environment data group corresponds to a data type. The definition of the data type is the same as before.

[0022] Optionally, in some embodiments of this application, the following data preprocessing steps are included after step S200: Step SS210: Select the category environment data group that meets the preset data type conditions from the above category environment data group set as the target environment data group.

[0023] In some embodiments, the execution entity may select a category environment data group that meets preset data type conditions from the aforementioned category environment data group set as the target environment data group. The preset data type conditions may be pre-defined categories environment data groups corresponding to data types that require preprocessing.

[0024] Step SS220: Preprocess each target environment data in the above target environment data group to generate a preprocessed environment data group and form a preprocessed classified environment data group set.

[0025] In some embodiments, the executing entity may preprocess each target environment data in the target environment data group to generate a preprocessed environment data group. The preprocessing may include, but is not limited to, at least one of data cleaning and data filling. The resulting preprocessed categorized environment data group can be used as the basis for subsequent data processing.

[0026] Step S300: Receive first data association information for the classified environment data set sent by the target terminal, and perform first association processing on each classified environment data in the classified environment data set based on the first data association information to generate the first associated environment data set.

[0027] In some embodiments, the execution entity may receive first data association information for the aforementioned categorized environmental data set sent by the target terminal, and perform a first association process on each categorized environmental data in the aforementioned categorized environmental data set based on the first data association information to generate a first associated environmental data set. The first data association information may be user-defined data association information of the target terminal.

[0028] By receiving the initial data association information customized by the target terminal user, preliminary association of the classified environmental data is achieved, laying the foundation for subsequent algorithm-based deep association. After the executing entity completes the classification processing of multimodal environmental data and generates classified environmental data sets, it enters the stage of receiving and associating the initial data association information. The target terminal can be a desktop computer, laptop computer operated by staff, or a mobile terminal device. Users can independently configure the initial data association information on the terminal's visual operation interface according to actual business needs. This information is essentially the data association rules set by the user based on business scenarios and analysis objectives, which can intuitively reflect the user's initial needs for the association relationship of multimodal environmental data.

[0029] The process of user-defined primary data association information is highly flexible and targeted, and can be illustrated with examples from actual application scenarios. For instance, in an urban environmental monitoring scenario, the categorized environmental data set includes a multispectral image set (image format), a meteorological dataset (text format), a laser point cloud data set (point cloud format), and a visible light video set (video format). If staff need to analyze the correlation between PM2.5 concentration in a certain area of ​​the city and the surrounding vegetation cover and meteorological conditions, they can make the following custom configuration on the target terminal's operation interface: First, select the subject of association as "PM2.5 concentration data in the meteorological data group", then select the objects of association in sequence as "band data representing vegetation cover in the multispectral image group", "wind speed data in the meteorological data group", and "three-dimensional data representing building height in the laser point cloud data group", and set the association conditions as "the same monitoring time period (e.g., May 1-May 7, 2024) + the same geographical area (e.g., the eastern CBD area of ​​the city, with a latitude and longitude range of 116.4°-116.5° east longitude and 39.9°-40.0° north latitude)", and configure the association priority as "vegetation cover data > wind speed data > building height data", and finally form a complete first data association information and send it to the server.

[0030] In some embodiments of this application, based on first data association information, a first association process is performed on each category environment data in the category environment data set to generate a first associated environment data set, specifically including steps S310 to S340: Step S310: Perform structured parsing on the received first data association information to extract the association subject, association object, association conditions, and association priority. The association conditions include time matching conditions and spatial matching conditions. The association subject is the core data corresponding to any category environment data group in the category environment data group set. The association object is the target data corresponding to other category environment data groups in the category environment data group set that have business association requirements with the association subject.

[0031] Step S320: Based on the parsed associated subject identifier, filter out the corresponding associated subject data from the classification environment data set. Based on the time matching condition and spatial matching condition in the association conditions, filter the data of the classification environment data set corresponding to each associated object to obtain candidate associated object data that meet the requirements of spatiotemporal consistency.

[0032] Step S330: According to the association priority in the first data association information, the candidate association object data and the association subject data are associated and bound in sequence to form an association data unit containing the association subject data, candidate association object data of each priority and association rule identifier.

[0033] Step S340: Integrate all associated data units that have been bound together to ensure that the data in each associated data unit meets the business association logic set by the first data association information, and all associated data units together constitute the first associated environment data set.

[0034] Based on the aforementioned custom first data association information, the execution entity's first association processing follows the logical progression of "rule parsing - data matching - association combination." First, upon receiving the first data association information, the execution entity performs structured parsing, extracting key association elements, including the association subject, association object, association conditions (time conditions, spatial conditions, etc.), association priority, and other core parameters, converting the unstructured rule description into executable data analysis instructions. Subsequently, based on the parsed association conditions, data filtering and matching are performed within the categorized environmental data set. For example, taking urban environmental monitoring, the execution entity first selects PM2.5 concentration data for the eastern CBD area of ​​the city from the meteorological data set for May 1st to May 7th, 2024, as the association subject data; then, it extracts vegetation cover band data for that time period and area from the multispectral image set, corresponding wind speed data from the meteorological data set, and corresponding building height data from the laser point cloud data set, as the association object data. During this process, timestamp alignment and geographic coordinate matching are used to ensure data consistency in the spatiotemporal dimensions, eliminating data that does not meet the association conditions. Finally, according to the association priority and the association logic set by the user, the data of the main association subject is combined with the data of each associated object to form a first association environmental data group with a preliminary association relationship. All these data groups together constitute the first association environmental data set. For example, PM2.5 concentration data at a certain time and location is combined with corresponding vegetation cover data, wind speed data, and building height data to form a first association environmental data group. This ensures that each data group is organically related to the analysis objectives set by the user, providing basic data that meets the user's needs for subsequent deep association based on relationship prediction algorithms.

[0035] Step S400: Based on the relationship prediction algorithm, determine the second data association information set corresponding to the first associated environment data set, and based on the second data association information set, merge the various first associated environment data sets included in the first associated environment data set to generate at least one merged environment data set, thus obtaining the merged environment data set.

[0036] In some embodiments, the execution entity may determine the second data association information set corresponding to the first associated environmental data set based on the relationship prediction algorithm, and merge each of the first associated environmental data sets included in the first associated environmental data set based on the second data association information set to generate at least one merged environmental data set, thereby obtaining a merged environmental data set.

[0037] In some embodiments, the aforementioned executing entity may determine the second data association information set corresponding to the first associated environmental data set based on a relationship prediction algorithm through the following steps. For each piece of first associated environment data in the aforementioned first associated environment data set, perform the following processing steps S410 to S450: Step S410: Determine the spatial relationship information between the aforementioned first associated environmental data and each first associated environmental data in the aforementioned first associated environmental data group after removing the aforementioned first associated environmental data, so as to generate a spatial relationship information set.

[0038] Step S420: Select each first associated environment data whose spatial relationship information satisfies the preset spatial conditions from the above first associated environment data set, and use them as second associated environment data to obtain the second associated environment dataset.

[0039] Optionally, after step S420, spatial alignment is performed on each of the second associated environment data in the second associated environment dataset to generate aligned second associated environment data as the second associated environment dataset.

[0040] Step S430: Select the first associated environment data whose corresponding time relationship information satisfies the preset time condition from the first associated environment data set after removing each second associated environment data set, and use them as the target associated environment data to obtain the target associated environment dataset. The preset time condition can be that the target associated environment data and the first associated environment data are within a preset time period.

[0041] Step S440: Merge the second associated environment dataset with the target associated environment dataset to generate a merged second associated environment dataset.

[0042] Step S450: Generate second data association information between the first associated environmental data and the second associated environmental dataset.

[0043] In some embodiments of this application, the following steps S460 to S480 can be used to merge the various first associated environment data groups included in the first associated environment data group set based on the second data association information set, so as to generate at least one merged environment data group and obtain a merged environment data group set: For each piece of second data association information in the aforementioned second data association information set, the following merging steps are performed: Step S460: Determine the first associated environment data corresponding to the second data association information as the root node.

[0044] Step S470: Based on the second data association relationship and the root node mentioned above, generate an environmental data association structure tree.

[0045] Step S480: Based on the above-mentioned environmental data association structure tree, merge the various second data association data corresponding to the above-mentioned second data association information to generate a merged environmental data group.

[0046] Step S500: Perform data analysis and processing on the merged environmental data set, and generate at least one environmental data map based on the data analysis results to obtain an environmental data map set.

[0047] In some embodiments, the aforementioned executing entity may perform data analysis processing on the aforementioned merged environmental data set, and generate at least one environmental data map based on the data analysis results, thereby obtaining an environmental data map set.

[0048] This step involves in-depth data analysis of the merged environmental data set after two rounds of association and merging. The analysis uncovers the inherent relationships and core characteristics between the data, generating an environmental data map that intuitively presents the data relationships. This provides a precise and effective data carrier for subsequent visualization. The merged environmental data set, as input for this step, has undergone dual processing—user-defined associations and algorithm-predicted associations—to eliminate relationship biases caused by hard associations, ensuring the authenticity and completeness of the relationships between data. Each merged environmental data set contains multiple types of environmental data with strong correlations, providing high-quality foundational data support for in-depth data analysis. The execution entity (such as a server) relies on a professional data analysis engine and map generation tools, proceeding step-by-step according to the logic of "data preprocessing optimization - multi-dimensional feature extraction - association pattern mining - map structure generation" to ensure the depth of data analysis and the intuitiveness of the map presentation.

[0049] In some embodiments of this application, data analysis processing is performed on the merged environmental data set, and at least one environmental data map is generated based on the data analysis results to obtain an environmental data map set, including steps S510 to S550: Step S510: Perform data preprocessing optimization on each merged environment data group in the merged environment data group set. The preprocessing optimization includes: converting the multi-format data in the merged environment data group into a unified analyzable format, and performing noise reduction processing through outlier removal, duplicate data deduplication and data smoothing algorithms to obtain a standardized data group.

[0050] Step S520: For different types of data in the standardized data set, appropriate feature extraction methods are used to extract features. For example, for image data (multispectral images, visible light video frames), texture, color, contour and specific environmental features are extracted by convolutional neural networks; for numerical data, mean, variance, trend and extreme value features are extracted by statistical analysis; for laser point cloud data, three-dimensional spatial features are extracted by point cloud segmentation and clustering algorithms to form a multi-dimensional feature set.

[0051] Step S530: Using association rule mining, time series analysis, and spatial correlation analysis algorithms, we can mine association patterns in a multi-dimensional feature set, determine the association type, association strength, and influence weight between features, and generate structured data analysis results. Among them, the association types include: positive correlation, negative correlation, causal relationship, etc.

[0052] Step S540: Construct an environmental data map based on the structured data analysis results: Map core data entities to map nodes. Nodes carry attribute information such as feature parameters and collection time. The size and color of nodes are set differently according to the importance or value of features. Construct connection edges between nodes according to the association type. The thickness and line type of the connection edges represent the association strength and type. The connection edges are labeled with association coefficients and influence weights.

[0053] Step S550: For different analysis objectives, generate corresponding environmental data maps. All maps are integrated to form an environmental data map set. Each map clearly corresponds to an analysis topic and data range.

[0054] When performing data analysis on merged environmental datasets, data preprocessing and optimization are first required to remove obstacles for subsequent analysis. Since the merged environmental datasets contain data in various formats such as video, images, text, and point clouds, the system first standardizes the data within each merged dataset, converting different types of data into a unified analyzable format. For example, laser point cloud data is converted into a 3D coordinate matrix, multispectral images are converted into pixel feature vectors, and meteorological text data is converted into structured numerical data. Simultaneously, further data noise reduction processing is performed to remove outliers and duplicate data that may have been introduced during the merging process, and smoothing algorithms are used to correct data fluctuations, ensuring data accuracy. Subsequently, the multi-dimensional feature extraction stage is entered, employing appropriate feature extraction methods for different types of data: For image data (multispectral images, visible light video frames), texture features, color features, and contour features are extracted using convolutional neural networks (CNNs), such as extracting key environmental features like vegetation index and water body boundaries from multispectral images; For numerical data such as meteorological and soil data, features like mean, variance, trend, and extreme values ​​are extracted using statistical analysis methods, such as calculating the monthly average temperature and humidity variation of a certain area; For laser point cloud data, three-dimensional spatial features such as terrain slope, building height, and vegetation coverage are extracted using point cloud segmentation and clustering algorithms.

[0055] After feature extraction is completed, the executing entity will use algorithms such as association rule mining, time series analysis, and spatial correlation analysis based on the relationships between data to uncover the inherent patterns between features. For example, it will analyze the positive correlation between vegetation index and soil moisture and sunshine duration, the correlation pattern between terrain slope and precipitation runoff, or the coupling relationship between a certain pollution indicator and the distribution of surrounding industries and meteorological conditions, etc., to form structured data analysis results that cover core information such as data feature parameters, correlation strength, and change patterns.

[0056] Based on the aforementioned in-depth data analysis results, the implementing entity will further generate an environmental data map, forming an environmental data map set. The generation of the environmental data map must follow the approach of "core element mapping - relationship visualization modeling - structured map output" to ensure that the map can clearly and accurately present the logical relationships between data. First, the implementing entity will map the core data entities from the data analysis results (such as "PM2.5 concentration," "vegetation coverage," "wind speed," and "topography slope") into nodes in the map. Each node will carry corresponding feature parameters (such as numerical range, unit, and collection time) as attribute information. The size and color of the nodes can be differentiated according to the importance of the features or the magnitude of the values. For example, the core data entities with the highest correlation strength will be set as larger nodes, and data entities with values ​​exceeding the standard will be marked as red nodes.

[0057] Secondly, based on the correlations obtained from data analysis (such as positive correlation, negative correlation, causal relationship, etc.), connection edges are constructed between nodes. The thickness and line type of the connection edges can be used to characterize the strength and type of correlation. For example, a thick solid line represents a strong correlation, a dashed line represents a weak correlation, and an arrow line represents a causal relationship. At the same time, key information such as correlation coefficient and influence weight are labeled on the connection edges to make the data relationships clear at a glance. For different analysis objectives and data scenarios, the executing entity will generate various types of environmental data maps. For example, in the urban environmental monitoring scenario, a correlation map of "pollution indicators-meteorological conditions-geographical features" and a time-series variation map of "vegetation cover-soil quality-hydrological distribution" are generated.

[0058] Finally, all generated environmental data maps are integrated to ensure that each map corresponds to a clear analysis topic and data range, forming a well-structured and hierarchical environmental data map set. This lays the foundation for subsequent visualization and display on target terminals, enabling users to quickly grasp the core correlations and changing patterns of multimodal environmental data through the maps, thereby improving the efficiency and accuracy of environmental data analysis.

[0059] Step S600: Send each environmental data map in the environmental data map set to the target terminal for visualization display.

[0060] In some embodiments, the aforementioned execution entity may send each environmental data map in the aforementioned environmental data map set to the aforementioned target terminal for visualization display.

[0061] from Figure 1As can be seen, the multimodal environmental data visualization method provided in this application avoids the generation of erroneous environmental data maps during environmental data analysis, thus preventing invalid visualization results. Specifically, the reason for generating erroneous environmental data maps during environmental data analysis and thus invalid visualization results is that multimodal environmental data often exhibits combined correlations. Simply associating multimodal data through hard data correlations leads to inconsistencies between the associated environmental data and the actual correlations, resulting in erroneous environmental data maps during environmental data analysis and thus invalid visualization results. Based on this, the multimodal environmental data visualization method provided in this application first obtains a multimodal environmental dataset. This allows for the acquisition of multimodal environmental data. Secondly, based on the various data types corresponding to the multimodal environmental dataset, the multimodal environmental data included in the dataset are classified to generate categorized environmental data sets. This allows for the classification of environmental data according to data type. Then, the system receives first data association information for the aforementioned categorized environmental data sets from the target terminal, and performs a first association process on each category of environmental data in the aforementioned categorized environmental data sets based on the first data association information to generate a first associated environmental data set. Thus, environmental data can be associated for the first time using the received association information. Next, based on a relationship prediction algorithm, the system determines second data association information corresponding to the aforementioned first associated environmental data set, and merges each first associated environmental data set included in the aforementioned first associated environmental data set based on the second data association information to generate at least one merged environmental data set. Thus, based on the predicted association information, the environmental data after the first association can be associated a second time and merged. Finally, the system performs data analysis processing on the aforementioned merged environmental data set, and generates at least one environmental data map based on the data analysis results to obtain an environmental data map set; each environmental data map in the aforementioned environmental data map set is sent to the aforementioned target terminal for visualization. Thus, an environmental data relationship map can be generated and visualized. Therefore, environmental data can be fully correlated through two associations, thus avoiding the generation of incorrect correlation graphs and invalid visualization results. In summary, this application effectively solves the problems of inconsistencies between correlations and reality, and invalid visualization results caused by traditional hard correlations of multimodal environmental data; it provides users with intuitive and effective data visualization support, significantly improving the analysis efficiency and application value of multimodal environmental data.

[0062] Figure 2 A schematic diagram of a multimodal environment data visualization device according to an embodiment of this application is shown. Figure 2As can be seen, the visualization device 200 for multimodal environment data includes: The acquisition unit 210 is used to acquire a multimodal environment dataset, wherein each multimodal environment data in the multimodal environment dataset corresponds to a unique data type; The classification unit 220 is used to classify the multimodal environment data in the multimodal environment dataset based on the data type, and generate a classified environment data set, wherein each classified environment data set corresponds to a data type. The association unit 230 is used to receive first data association information for the classification environment data set sent by the target terminal, and to perform first association processing on each classification environment data in the classification environment data set based on the first data association information to generate the first associated environment data set. The merging unit 240 is used to determine the second data association information set corresponding to the first associated environment data set based on the relationship prediction algorithm, and to merge each of the first associated environment data sets included in the first associated environment data set based on the second data association information set, so as to generate at least one merged environment data set and obtain a merged environment data set. Analysis unit 250 is used to perform data analysis and processing on the merged environmental data set, and to generate at least one environmental data map based on the data analysis results, thereby obtaining an environmental data map set.

[0063] Display unit 260 is used to send each environmental data map in the environmental data map set to the target terminal for visualization display.

[0064] It should be noted that the aforementioned visualization devices for multimodal environmental data can implement the aforementioned visualization methods for multimodal environmental data one by one. For implementation details, please refer to the aforementioned content, which will not be repeated here.

[0065] Figure 3 This invention illustrates a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external devices via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a method for visualizing multimodal environmental data.

[0066] In one embodiment, the electronic device provided in this application includes a memory and a processor. The memory stores a database and a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the aforementioned method for visualizing multimodal environment data.

[0067] The above is as stated in this application. Figure 2 The method executed by the visualization device for multimodal environmental data disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0068] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the aforementioned method for visualizing multimodal environment data.

[0069] It should be noted that the functions or steps that the above-mentioned electronic devices or computer-readable storage media can achieve can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0070] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0071] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0072] The above-described embodiments are only used to illustrate the technical application of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical applications described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical applications to deviate from the spirit and scope of the technical applications of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of visualizing multi-modal environmental data, characterized in that, include: Obtain a multimodal environment dataset, wherein each multimodal environment data in the dataset corresponds to a unique data type; Based on the data type, the multimodal environment data in the multimodal environment dataset is classified to generate a categorized environment data set, where each categorized environment data set corresponds to a data type. The system receives first data association information for a categorized environmental data set sent by the target terminal, and performs first association processing on each categorized environmental data in the categorized environmental data set based on the first data association information to generate a first associated environmental data set. Based on the relationship prediction algorithm, the second data association information set corresponding to the first associated environment data set is determined, and based on the second data association information set, the various first associated environment data sets included in the first associated environment data set are merged to generate at least one merged environment data set, thus obtaining the merged environment data set. Data analysis and processing are performed on the merged environmental data set, and at least one environmental data map is generated based on the data analysis results to obtain an environmental data map set; Each environmental data map in the environmental data map set is sent to the target terminal for visualization.

2. The method of claim 1, wherein, Obtaining a multimodal environment dataset includes: Receive at least one satellite remote sensing image of the target area transmitted by associated satellite equipment, as a multispectral image set; The target meteorological sensor is controlled to collect meteorological data of the target area and generate a meteorological dataset. Control the associated unmanned aerial vehicle to acquire laser point cloud data and visible light video of the target area, and generate laser point cloud dataset and visible light video; The multispectral image set, the meteorological dataset, the laser point cloud dataset, and the visible light video are combined into a multimodal environment dataset.

3. The method of claim 1, wherein, After the step of generating the categorized environmental data set, the method further includes: Select a category environment data group that meets the preset data type conditions from the category environment data group set and use it as the target environment data group; Each target environment data in the target environment data group is preprocessed to generate a preprocessed environment data group, forming a preprocessed classified environment data group set.

4. The method of claim 1, wherein, Based on the first data association information, a first association process is performed on each category environment data in the category environment data set to generate a first associated environment data set, including: The received first data association information is structured and parsed to extract the association subject, association object, association conditions and association priority. The association conditions include time matching conditions and spatial matching conditions. Based on the associated entity identifier obtained from the parsing, the corresponding associated entity data is selected from the classification environment data set. Based on the time matching condition and spatial matching condition in the association condition, the data of the classification environment data set corresponding to each associated object is filtered to obtain candidate associated object data that meet the requirements of spatiotemporal consistency. According to the association priority, the candidate association object data and the association subject data are sequentially associated and bound to form an association data unit containing the association subject data, the candidate association object data of each priority, and the association rule identifier; All associated data units are integrated to ensure that the data within each associated data unit meets the business association logic set by the first data association information, and all associated data units together constitute the first associated environment data set.

5. The method of claim 1, wherein, Based on the relationship prediction algorithm, the second data association information set corresponding to the first associated environment data set is determined, including: For each piece of first associated environment data in the first associated environment data set, perform the following processing steps: Determine the spatial relationship information of each first associated environmental data in the first associated environmental data group after removing the first associated environmental data, so as to generate a spatial relationship information set; Select each first associated environment data whose spatial relationship information satisfies the preset spatial conditions from the first associated environment data set and use it as the second associated environment data to obtain the second associated environment dataset; Select the first associated environment data whose corresponding time relationship information meets the preset time conditions from the first associated environment data set after removing each second associated environment data, and use them as the target associated environment data to obtain the target associated environment dataset; The second associated environment dataset is merged with the target associated environment dataset to generate a merged second associated environment dataset. Generate second data association information between the first associated environment data and the second associated environment dataset.

6. The method of claim 1, wherein, Based on the second data association information set, the various first associated environment data groups included in the first associated environment data group set are merged to generate at least one merged environment data group, resulting in a merged environment data group set, including: For each piece of second data association information in the second data association information set, perform the following steps: The first associated environment data corresponding to the second data association information is determined as the root node; Based on the second data association relationship and the root node, an environmental data association structure tree is generated; Based on the environmental data association structure tree, the various second data association data corresponding to the second data association information are merged to generate a merged environmental data group.

7. The method of claim 1, wherein, The merged environmental data set is subjected to data analysis and processing, and based on the data analysis results, at least one environmental data map is generated to obtain an environmental data map set, including: For each merged environment data group in the merged environment data group set, data preprocessing optimization is performed. The preprocessing optimization includes: converting the multi-format data in the merged environment data group into a unified analyzable format, and performing noise reduction processing through outlier removal, duplicate data deduplication and data smoothing algorithms to obtain a standardized data group. Adaptive feature extraction methods are used to extract features from different types of data in a standardized dataset. The association rule mining, time series analysis, and spatial correlation analysis algorithms are used to mine association patterns in multi-dimensional feature sets, determine the association type, association strength, and influence weight between features, and generate structured data analysis results. Environmental data mapping is constructed based on the results of structured data analysis. For different analytical objectives, corresponding environmental data maps are generated, and all maps are integrated to form an environmental data map set.

8. A device for visualizing multi-modal environmental data, characterized in that, The device includes: The acquisition unit is used to acquire a multimodal environment dataset, wherein each multimodal environment data in the multimodal environment dataset corresponds to a unique data type; A classification unit is used to classify the multimodal environment data in the multimodal environment dataset based on the data type, and generate a classification environment data set, wherein each classification environment data set corresponds to a data type. The association unit is used to receive first data association information for the classification environment data set sent by the target terminal, and to perform first association processing on each classification environment data in the classification environment data set based on the first data association information to generate the first associated environment data set. The merging unit is used to determine the second data association information set corresponding to the first associated environment data set based on the relationship prediction algorithm, and to merge each of the first associated environment data sets included in the first associated environment data set based on the second data association information set, so as to generate at least one merged environment data set and obtain a merged environment data set. The analysis unit is used to perform data analysis and processing on the merged environmental data set, and to generate at least one environmental data map based on the data analysis results, thereby obtaining an environmental data map set. The display unit is used to send each environmental data map in the environmental data map set to the target terminal for visualization.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for visualizing multimodal environment data as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by a processor, it implements the steps of the method for visualizing multimodal environment data as described in any one of claims 1 to 7.

Images