A system and method for emergency data standardization based on data virtualization

Through the emergency data standardization system based on data virtualization, the multi-dimensional incremental synchronization and consistency guarantee algorithm is used to solve the data inconsistency and integrity problems in emergency data management, and the efficient processing and unified view of data are realized, and emergency support is supported. Decision making and response.

CN118673182BActive Publication Date: 2024-11-05北京紫云智能科技有限公司
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411140134.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-05
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

The existing technology has problems with data inconsistency and integrity in emergency data management, especially in the multi-source data synchronization process, which leads to impaired data reliability and accuracy, and traditional data integration methods are difficult to implement under complex network conditions. Real-time or timely synchronization can easily cause data security concerns.

Method used

Adopt emergency data standardization system based on data virtualization, including metadata extraction module, heterogeneous data fusion module, data standardization module, mapping preprocessing module and data mapping layer module, through multi-dimensional incremental synchronization algorithm and consistency guarantee algorithm , ensure data consistency and accuracy between the data source and the virtualized system, and map standardized data to the original data source through the data mapping layer.

Benefits of technology

It effectively solves data inconsistency and integrity problems, improves data processing efficiency, provides a unified data view, supports decision-making and emergency response, and ensures data availability and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118673182B_ABST
    Figure CN118673182B_ABST
Patent Text Reader

Abstract

The present invention provides an emergency data standardization system and method based on data virtualization, including: metadata extraction module, heterogeneous data fusion module, data standardization module, mapping pre-processing module and data mapping layer module, through metadata extraction, data fusion, data standardization, intelligent incremental synchronization and consistency guarantee steps, to ensure the consistency, accuracy and availability of data. These steps not only improve the efficiency of data processing, but also provide a unified data view for upper-layer applications, which is helpful for decision support and rapid response in emergency situations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an emergency data standardization system and method based on data virtualization. Background Art

[0002] With the rapid development and popularization of information technology, data virtualization and data standardization are two important technical trends in the field of modern data management. They have experienced significant development and are combined with each other in many scenarios to provide more powerful data processing capabilities. Among them, data virtualization, as an advanced data integration technology, allows applications to access and operate data from multiple heterogeneous data sources in a unified manner without worrying about the actual location or format of the data; in addition, data standardization refers to converting data into a consistent format or structure to ensure the comparability and consistency of the data. With the rise of big data and artificial intelligence technologies, data standardization has become particularly important.

[0003] Although data virtualization and data standardization technologies have played an important role in the field of emergency data management, there are still some challenges and defects. Different institutions or departments may adopt different data standards and specifications, resulting in poor interoperability of data between different systems. The lack of a unified data standard framework has greatly increased the workload of data standardization. During the data synchronization process, multiple data sources may modify the same data at the same time, resulting in data conflicts. The lack of an effective conflict resolution mechanism may lead to damage to data consistency and accuracy.

[0004] What is even more difficult is that when using traditional data integration methods to solve the above problems, many practical difficulties will be encountered. Traditional methods require real-time or scheduled synchronization of multi-source data, but under complex and changeable network conditions and facing a variety of data source systems, the implementation of data synchronization solutions becomes extremely difficult and easily leads to concerns about data security. Delays, errors or losses that may occur during data synchronization further exacerbate data inconsistency and integrity issues, affecting the reliability of clinical decision-making. Summary of the invention

[0005] The embodiment of the present invention provides an emergency data standardization system and method based on data virtualization, which can at least solve the technical problems of data inconsistency and integrity in the prior art, and affect the reliability of clinical decision-making.

[0006] In a first aspect of an embodiment of the present invention, there is provided an emergency data standardization system based on data virtualization, comprising: a metadata extraction module, a heterogeneous data fusion module, a data standardization module, a mapping pre-processing module and a data mapping layer module, wherein:

[0007] The metadata extraction module extracts metadata through multiple different data source interfaces and sends the metadata to the virtualization system as a basis for data view and virtual table definition, wherein the metadata includes multiple types of heterogeneous emergency data;

[0008] The heterogeneous data fusion module receives the metadata of the metadata extraction module, performs data fusion through the data virtualization server, and determines the fused data so that the upper layer application can access multiple types of heterogeneous data;

[0009] The data standardization module standardizes the fused data based on a preset emergency data standard to obtain emergency data;

[0010] The mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; when multiple data sources are detected to modify the same data, the consistency and accuracy of the data is ensured by a consistency assurance algorithm;

[0011] The data mapping layer module maps the emergency data obtained after the standardization process to the original data source according to the mapping from the virtual table to the data source.

[0012] Furthermore, the heterogeneous data fusion module includes:

[0013] A data source identification unit, used to identify the type of each data source connected to the data virtualization server;

[0014] The data adapter unit configures a data adapter for each type of data source to parse the structure and content of the data source to convert metadata in different formats into a unified intermediate representation;

[0015] A data mapping unit analyzes the type of metadata to establish a mapping relationship between the data source and the internal data model of the virtualization system, so that the metadata can be converted between different data sources and the virtualization system;

[0016] The data fusion unit receives the intermediate representation data converted by the data adapter unit and executes the data fusion algorithm to create a data view so that the upper-layer application can access the data from different data sources with a unified interface.

[0017] Furthermore, the data fusion unit includes:

[0018] Obtaining the number of records of the data source in the metadata extraction module to determine the integrity of the data source;

[0019] Determining the consistency of the data source based on the mapping relationship between the data source established by the data mapping unit and the internal data model of the virtualization system;

[0020] Determining the timeliness of data source update based on data source update events in the metadata extraction module;

[0021] Determining the accuracy of the data source based on the emergency data and metadata outputted from the data standardization module;

[0022] Obtaining the data source and preset measurement indicators in the metadata extraction module to determine the reliability of the data source;

[0023] Based on the data mapping layer module, the standardized emergency data is mapped to the original data source to determine the richness of the data source;

[0024] Calculate the overall quality score of the data source based on the consistency, timeliness, accuracy, reliability and richness of the data source;

[0025] When the comprehensive quality score is greater than a preset value, an adaptive data fusion strategy is used to construct a data view.

[0026] Furthermore, the mapping pre-processing module includes: a data synchronization unit and a conflict resolution unit;

[0027] After detecting a data source change event, the data synchronization unit first executes a multi-dimensional incremental synchronization algorithm to synchronize the changed data to the virtualization system;

[0028] During the synchronization process, the data synchronization unit records the version number and timestamp of each data item as synchronization information for use by the conflict resolution unit;

[0029] After the synchronization is completed, the data synchronization unit transmits the synchronization information to the conflict resolution unit;

[0030] After receiving the synchronization information, the conflict resolution unit checks whether there is a data conflict;

[0031] If a data conflict is detected, the conflict resolution unit uses a consistency guarantee algorithm to resolve the conflict;

[0032] The conflict resolution unit returns the resolved data version to the data synchronization unit, and the data synchronization unit updates the data in the virtualization system accordingly;

[0033] The data synchronization unit and the conflict resolution unit communicate with each other through a message passing mechanism to ensure the coordination and consistency of the synchronization process and the conflict resolution process.

[0034] Furthermore, the multi-dimensional incremental synchronization algorithm includes:

[0035] Given a data item D i Historical change records of H i , data source S j Characteristics of C j , the importance of data items W i , the quality score Q of the data source j , and the data source credibility T j , predict data item D i The predicted probability P of a change occurring in the next time period i for:

[0036] ;

[0037] in, is the sigmoid function, which is used to convert the prediction results into probability values. Indicates extracting historical records H through time series model i The characteristic vector of Represents the data source characteristics C obtained through cluster analysis j The characteristic vector of Represents the importance of the data item predicted by the prediction model W i The characteristic vector of Indicates the quality score Q of the data source j The nonlinear weight function, Indicates the credibility of the data source T j The characteristic vector of , , , , is the weight coefficient, which is used to adjust the influence of each factor;

[0038] For data items with a predicted probability greater than a threshold, data synchronization is performed in the previous time period of the preset time node. For data items with a predicted probability less than a threshold, data synchronization is performed in the next time period of the preset time node to reduce data redundancy.

[0039] Furthermore, the data mapping layer module includes:

[0040] A type determination unit, used to determine the types of each data source in the metadata extraction module, and determine a packaging table based on the data source type to parse the emergency data;

[0041] A virtual table construction unit, used to define a virtual table based on a preset encapsulation table, wherein the virtual table is determined according to the logical structure of the emergency data;

[0042] An execution unit, used for executing mapping rules to convert the emergency data in the data source into a format of a virtual table;

[0043] The query unit is used to establish a query port to enable upper layer data applications to access emergency data.

[0044] Furthermore, when emergency data conversion is involved in the virtual table construction unit in the data mapping layer module, Box-Cox transformation is used to perform data conversion to improve the normality of data distribution, wherein the step of using Box-Cox transformation to perform data conversion to improve the normality of data distribution includes:

[0045] The maximum likelihood estimation method is used to determine the optimal λ value, where the λ value is used to determine the type of transformation;

[0046] For each non-negative emergency data value, if the λ value is not equal to zero, the power transformation method is used to calculate the λ power of the original data value, subtract one from the result of the λ power, and divide the result by the λ value;

[0047] If the λ value is equal to zero, the logarithmic transformation method is used to take the natural logarithm of the original data value;

[0048] After applying the transformation, normality test methods were used to verify whether the transformed data were closer to a normal distribution;

[0049] If the transformed data verified normality, standardization was performed.

[0050] In a second aspect, an embodiment of the present invention further provides an emergency data standardization method based on data virtualization, including:

[0051] A control metadata extraction module extracts metadata through multiple different data source interfaces and sends the metadata to a virtualization system to serve as a basis for defining data views and virtual tables, wherein the metadata includes multiple types of heterogeneous emergency data;

[0052] Controlling the heterogeneous data fusion module to receive the metadata of the metadata extraction module, performing data fusion through the data virtualization server, and determining fused data so that upper layer applications can access multiple types of heterogeneous data;

[0053] The control data standardization module standardizes the fused data based on the preset emergency data standard to obtain emergency data;

[0054] The control mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; when multiple data sources are detected to modify the same data, the consistency and accuracy of the data is ensured through a consistency assurance algorithm;

[0055] The control data mapping layer module maps the emergency data obtained after standardization to the original data source according to the mapping of the virtual table to the data source.

[0056] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute an emergency data standardization method based on data virtualization as described in the second aspect.

[0057] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes an emergency data standardization method based on data virtualization as described in the second aspect.

[0058] An emergency data standardization system based on data virtualization provided by an embodiment of the present invention comprises: a metadata extraction module, a heterogeneous data fusion module, a data standardization module, a mapping pre-processing module and a data mapping layer module, wherein: the metadata extraction module extracts metadata through multiple different data source interfaces, and sends the metadata to a virtualization system as a basis for data view and virtual table definition, and the metadata contains multiple types of heterogeneous emergency data; the heterogeneous data fusion module receives the metadata of the metadata extraction module, performs data fusion through a data virtualization server, and determines fused data so that upper-layer applications can access multiple types of heterogeneous data; the data standardization module standardizes the fused data based on a preset emergency data standard to obtain emergency data; the mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is monitored, the emergency data in the virtualization system is ensured to be consistent with the data source through a multi-dimensional incremental synchronization algorithm; when it is monitored that multiple data sources modify the same data, the consistency and accuracy of the data are ensured through a consistency assurance algorithm; the data mapping layer module maps the emergency data obtained after standardization to the original data source according to the mapping of the virtual table to the data source. The emergency data standardization system based on data virtualization can effectively process heterogeneous emergency data from different data sources, and ensure the consistency, accuracy and availability of data through metadata extraction, data fusion, data standardization, intelligent incremental synchronization and consistency assurance steps. These steps not only improve the efficiency of data processing, but also provide a unified data view for upper-level applications, which is helpful for decision support and rapid response in emergency situations.

[0059] The beneficial effects of the embodiments of the present invention can refer to the technical effects corresponding to the technical features in the specific implementation methods, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a module of an emergency data standardization system based on data virtualization provided by an embodiment of the present invention;

[0061] Figure 2 A flowchart of another method for emergency data standardization based on data virtualization is provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0064] First, the present invention provides an emergency data standardization system based on data virtualization, such as Figure 1 As shown, it includes: metadata extraction module 01, heterogeneous data fusion module 02, data standardization module 03, mapping pre-processing module 04 and data mapping layer module 05, wherein:

[0065] The metadata extraction module 01 extracts metadata through multiple different data source interfaces and sends the metadata to the virtualization system as a basis for data view and virtual table definition, wherein the metadata includes multiple types of heterogeneous emergency data;

[0066] The heterogeneous data fusion module 02 receives the metadata of the metadata extraction module 01, performs data fusion through the data virtualization server, and determines the fused data so that the upper layer application can access multiple types of heterogeneous data;

[0067] The data standardization module 03 performs standardization processing on the fused data based on the preset emergency data standard to obtain emergency data;

[0068] The mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; when multiple data sources are detected to modify the same data, the consistency and accuracy of the data is ensured by a consistency assurance algorithm;

[0069] The data mapping layer module 05 maps the emergency data obtained after the standardization process to the original data source according to the mapping from the virtual table to the data source.

[0070] In a complete embodiment, it is assumed that emergency data comes from multiple data sources such as meteorological stations, earthquake monitoring stations, river water level monitoring stations, social media, and drone monitoring. The metadata extraction module extracts metadata from these systems, including data table names, field names, data types, primary keys, and other information. The metadata extraction module can accurately collect metadata information from different data sources, providing a basis for subsequent data fusion and standardization; the heterogeneous data fusion module receives the metadata provided by the metadata extraction module and uses the data virtualization server to perform data fusion. For example, the weather forecast data of the meteorological station is fused with the earthquake warning information of the earthquake monitoring station to form a unified data view. Through data fusion, the upper-level application can access the unified, fused data view, which greatly improves the convenience of data access and the efficiency of data processing; the data standardization module standardizes the fused data according to the preset emergency data standards. For example, the timestamps of all meteorological data are unified into the UTC time format, and the geographic location coordinates of different monitoring stations are unified into the WGS84 coordinate system. Data standardization ensures the consistency and comparability of data, improves data quality, and is conducive to subsequent data analysis and decision support. Assuming that the weather forecast data of the meteorological station is monitored to change, the mapping pre-processing module will immediately start the synchronization process and use an intelligent incremental synchronization algorithm. This algorithm comprehensively considers multi-dimensional factors such as historical change records, data source characteristics, data item importance, data source quality score, and data source credibility to predict the possibility of data item changes. When multiple data sources are monitored to modify the same data, a consistency assurance algorithm is used. This algorithm resolves data conflicts through data version control, timestamp mechanism, priority strategy, etc. to ensure data consistency and accuracy. Through multi-dimensional incremental synchronization algorithms and consistency assurance algorithms, the efficiency and accuracy of data synchronization are ensured, data redundancy and delay problems are reduced, and the stability and reliability of the system are improved. The data mapping layer module maps the standardized emergency data back to the original data source according to the mapping relationship between the virtual table and the data source. For example, the standardized weather forecast data is mapped back to the meteorological station. Through the data mapping layer module, a two-way mapping between the standardized data and the original data source is achieved, which facilitates data updating and maintenance and improves data availability.

[0071] In addition, emergency data generally refers to data collected and used in emergency situations to help quickly respond to and manage crises. This type of data can cover multiple areas, including but not limited to public safety, natural disaster response, infrastructure damage assessment, etc. Medical data can be considered part of emergency data in some cases, especially when it comes to public health emergencies (such as epidemic outbreaks), mass casualty events, or other situations that require emergency medical intervention. In this scenario, medical data may include key information such as patient information, disease transmission patterns, available medical resources (such as the number of beds, the number of medical personnel), etc., which are critical for effectively managing and responding to emergencies. Based on this, an embodiment of the present invention also provides an embodiment:

[0072] The metadata extraction module extracts metadata through multiple different data source interfaces and sends the metadata to the virtualization system as a basis for data view and virtual table definition, wherein the metadata includes multiple types of heterogeneous emergency data;

[0073] The module extracts metadata through multiple different data source interfaces and sends the metadata to the virtualization system as a basis for data view and virtual table definition, wherein the metadata includes multiple types of heterogeneous emergency data.

[0074] Example: Assume that emergency data comes from hospital information system (HIS), laboratory information system (LIS), ambulance dispatch system, etc. The metadata extraction module extracts metadata from these systems, including data table name, field name, data type, primary key and other information.

[0075] Beneficial effects: The metadata extraction module can accurately collect metadata information from different data sources, providing a basis for subsequent data fusion and standardization.

[0076] The heterogeneous data fusion module receives the metadata of the metadata extraction module, performs data fusion through the data virtualization server, and determines the fused data so that the upper layer application can access multiple types of heterogeneous data;

[0077] The module receives metadata from the metadata extraction module, performs data fusion through a data virtualization server, and determines fused data so that upper-layer applications can access multi-type heterogeneous data.

[0078] Embodiment: The heterogeneous data fusion module receives the metadata provided by the metadata extraction module and uses the data virtualization server to perform data fusion, for example, fusion of basic patient information in the hospital information system and laboratory test results in the laboratory information system.

[0079] Beneficial effects: Through data fusion, upper-level applications can access a unified, fused data view, greatly improving the convenience of data access and the efficiency of data processing.

[0080] The data standardization module standardizes the fused data based on a preset emergency data standard to obtain emergency data;

[0081] This module standardizes the fused data based on the preset emergency data standards to obtain emergency data.

[0082] Example: The data standardization module standardizes the fused data according to the preset emergency data standard. For example, the age fields of all patients are standardized into the same format (such as YYYY-MM-DD), and the laboratory test results are standardized according to the international system of units.

[0083] Beneficial effects: Data standardization ensures the consistency and comparability of data, improves data quality, and facilitates subsequent data analysis and decision support.

[0084] The mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; when multiple data sources are detected to modify the same data, the consistency and accuracy of the data is ensured by a consistency assurance algorithm;

[0085] This module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the above data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; when it is detected that multiple data sources have modified the same data, the consistency and accuracy of the data is ensured through a consistency assurance algorithm.

[0086] Example:

[0087] Monitor change events of data sources: Assuming that a patient's record in the hospital information system is updated, the mapping pre-processing module will immediately start the synchronization process.

[0088] Multi-dimensional incremental synchronization algorithm: Use an intelligent incremental synchronization algorithm that comprehensively considers multi-dimensional factors such as historical change records, data source characteristics, data item importance, data source quality score, and data source credibility to predict the possibility of change of data items.

[0089] Consistency assurance algorithm: When it is detected that multiple data sources modify the same data, a consistency assurance algorithm is used. This algorithm resolves data conflicts through data version control, timestamp mechanism, priority strategy, etc. to ensure data consistency and accuracy.

[0090] Beneficial effects: Through the multi-dimensional incremental synchronization algorithm and consistency guarantee algorithm, the efficiency and accuracy of data synchronization are ensured, data redundancy and delay problems are reduced, and the stability and reliability of the system are improved.

[0091] The data mapping layer module maps the emergency data obtained after the standardization process to the original data source according to the mapping from the virtual table to the data source.

[0092] This module maps the emergency data obtained after standardization to the original data source based on the mapping of virtual tables to data sources.

[0093] Embodiment: The data mapping layer module maps the standardized emergency data back to the original data source according to the mapping relationship between the virtual table and the data source. For example, the standardized basic information of the patient is mapped back to the hospital information system.

[0094] Beneficial effects: Through the data mapping layer module, a two-way mapping between standardized data and original data sources is achieved, which facilitates data updating and maintenance and improves data availability.

[0095] Different data sources may have different data structures and formats. Therefore, first identifying the data source type is a prerequisite for data adaptation and fusion. At the same time, since different data sources may use different data formats and encodings, an adapter is needed to unify these differences to facilitate data fusion. Based on this, the embodiment of the present invention provides a heterogeneous data fusion module, including:

[0096] A data source identification unit, used to identify the type of each data source connected to the data virtualization server;

[0097] The data source identification unit is capable of automatically identifying various data source types connected to the data virtualization server, such as weather stations, earthquake monitoring stations, river water level monitoring stations, social media APIs, etc.

[0098] Beneficial effects: By automatically identifying the data source type, the subsequent data adaptation and fusion process can be simplified, and the flexibility and scalability of the system can be improved.

[0099] The data adapter unit configures a data adapter for each type of data source to parse the structure and content of the data source to convert metadata in different formats into a unified intermediate representation;

[0100] A data adapter is configured for each type of data source to parse the structure and content of the data source and convert metadata in different formats into a unified intermediate representation. For example, a weather station data adapter: the data obtained from the weather station may be in XML or JSON format, and the data adapter parses it and converts it into a unified intermediate representation. For example, an earthquake monitoring station data adapter: the data from the earthquake monitoring station may be stored in CSV format, and the data adapter parses it and converts it into a unified intermediate representation.

[0101] Beneficial effects: The data adapter unit can process data sources in different formats and convert them into a unified intermediate representation to facilitate subsequent data fusion and processing.

[0102] A data mapping unit analyzes the type of metadata to establish a mapping relationship between the data source and the internal data model of the virtualization system, so that the metadata can be converted between different data sources and the virtualization system;

[0103] The data mapping unit is used to analyze the type of metadata, such as timestamp, geographic location, measurement value, etc., and then establish a mapping relationship, such as mapping the timestamp field of the meteorological station and the timestamp field of the earthquake monitoring station to a unified timestamp field in the virtualization system.

[0104] Beneficial effects: By establishing a mapping relationship, it can be ensured that the data between different data sources can be correctly matched, which is convenient for subsequent data fusion and analysis.

[0105] The data fusion unit receives the intermediate representation data converted by the data adapter unit and executes the data fusion algorithm to create a data view so that the upper-layer application can access the data from different data sources with a unified interface.

[0106] It receives intermediate representation data from the data adapter unit, such as timestamp, temperature, humidity and other data from the weather station, as well as timestamp, magnitude, focal depth and other data from the earthquake monitoring station, executes the data fusion algorithm, and integrates these data into a unified data view. The created data view allows upper-level applications to access these data with a unified interface.

[0107] Beneficial effects: The data fusion unit creates a unified data view, allowing upper-layer applications to easily access and process data from different data sources, thereby improving data availability and application development efficiency.

[0108] In practical applications, if a region encounters a serious natural disaster, such as a flood or earthquake, the emergency response team needs to obtain timely weather forecasts, river water levels, earthquake activity and other relevant information in order to make correct decisions. At this time, the emergency data standardization system based on data virtualization plays a key role.

[0109] Beneficial effects:

[0110] Improved data availability: The data adapter unit converts data in different formats into a unified intermediate representation, so that the data fusion unit can easily integrate the data into a unified data view, thereby improving data availability.

[0111] Simplified data access: By creating a unified data view, upper-level applications can easily access and process data from different data sources, improving the efficiency of application development.

[0112] Enhanced system flexibility and scalability: The data source identification unit can automatically identify various data source types, simplifying the subsequent data adaptation and fusion process and improving the system flexibility and scalability.

[0113] Ensure data consistency and accuracy: By establishing a mapping relationship, ensure that data between different data sources can be correctly matched, thereby improving data consistency and accuracy.

[0114] In another embodiment provided by the present invention, the data fusion unit includes:

[0115] Obtain the number of records of the data source in the metadata extraction module to determine the integrity of the data source:

[0116] In this step, the integrity of the data source is evaluated by obtaining the number of records of the data source in the metadata extraction module.

[0117] Embodiment: The data fusion unit obtains the number of records of each data source from the metadata extraction module, such as the number of weather forecast records obtained from a meteorological station, the number of earthquake activity records obtained from an earthquake monitoring station, and the like.

[0118] Beneficial effect: By evaluating the integrity of the data source, you can ensure that the data provided by the data source is complete and no important information is omitted.

[0119] Determine the consistency of the data source based on the mapping relationship between the data source established by the data mapping unit and the internal data model of the virtualization system:

[0120] In this step, the consistency of the data source is evaluated by analyzing the mapping relationship between the data source and the internal data model of the virtualization system.

[0121] Embodiment: The data fusion unit checks whether the data in the data source matches the internal data model of the virtualization system according to the mapping relationship established by the data mapping unit, for example, checks whether the timestamp field of the earthquake monitoring station is consistent with the timestamp field in the virtualization system.

[0122] Beneficial effects: Ensuring consistency between the data source and the internal data model of the virtualization system can improve the accuracy and reliability of the data.

[0123] Based on the data source update event in the metadata extraction module, determine the timeliness of the data source update:

[0124] In this step, the timeliness of data source update is evaluated by monitoring data source update events.

[0125] Embodiment: The data fusion unit monitors the update events of the data source in the metadata extraction module, for example, monitoring whether the weather forecast data of the weather station is updated within a specified time.

[0126] Benefits: By evaluating the timeliness of data source updates, you can ensure that the data is up-to-date and valid, which is very important for emergency response.

[0127] Determine the accuracy of the data source based on the emergency data and metadata output from the data standardization module:

[0128] In this step, the accuracy of the data source is evaluated by comparing the emergency data and metadata output by the data standardization module.

[0129] Embodiment: The data fusion unit compares the emergency data output by the data standardization module (eg, standardized weather forecast data) with the metadata provided by the metadata extraction module (eg, original data from a weather station) to check whether there is any difference.

[0130] Beneficial effects: Ensuring the accuracy of data sources can improve the reliability and effectiveness of emergency response decisions.

[0131] Obtain the data source and preset measurement indicators in the metadata extraction module to determine the reliability of the data source:

[0132] In this step, the reliability of the data source is evaluated by comparing the data provided by the data source with the preset measurement indicators.

[0133] Embodiment: The data fusion unit obtains data provided by the data source in the metadata extraction module, and compares the data with preset measurement indicators (such as data update frequency, data integrity, etc.).

[0134] Beneficial effects: Ensuring the reliability of data sources can improve the trust in emergency response decisions.

[0135] Based on the data mapping layer module, the standardized emergency data is mapped to the original data source to determine the richness of the data source:

[0136] In this step, the richness of the data source is evaluated by mapping the normalized emergency data back to the original data source.

[0137] Embodiment: The data fusion unit maps the standardized emergency data (eg, standardized river water level data) back to the original data source (eg, river water level monitoring station) to check whether all necessary data fields are covered.

[0138] Beneficial effect: Ensuring the richness of data sources can improve the integrity and value of data.

[0139] The overall quality score of the data source is calculated based on the consistency, timeliness, accuracy, reliability and richness of the data source:

[0140] In this step, the consistency, timeliness, accuracy, reliability, and richness of the data sources are comprehensively evaluated, and a comprehensive quality score is calculated.

[0141] Embodiment: The data fusion unit calculates the comprehensive quality score of the data source by using weighted average or other statistical methods according to the above evaluation results.

[0142] Beneficial effects: By calculating the comprehensive quality score, the overall quality level of the data source can be quantified, providing a basis for subsequent data fusion strategies.

[0143] When the comprehensive quality score is greater than a preset value, an adaptive data fusion strategy is used to construct a data view:

[0144] In this step, when the comprehensive quality score of the data source exceeds the preset value, an adaptive data fusion strategy is used to construct a data view.

[0145] Embodiment: If the comprehensive quality score of the data source exceeds a preset value (eg, 80 points), the data fusion unit adopts an adaptive data fusion strategy to fuse data from different data sources into a unified data view.

[0146] Beneficial effects: Building data views through adaptive data fusion strategies can ensure that upper-level applications can access data from different data sources with a unified interface, improving data availability and application development efficiency.

[0147] A specific embodiment of a mapping pre-processing module in an emergency data standardization system based on data virtualization, taking the field of natural disaster emergency response as an example, the mapping pre-processing module includes: a data synchronization unit and a conflict resolution unit;

[0148] After detecting a data source change event, the data synchronization unit first executes a multi-dimensional incremental synchronization algorithm to synchronize the changed data to the virtualization system;

[0149] After detecting a data source change event, first execute a multi-dimensional incremental synchronization algorithm to synchronize the changed data to the virtualization system and monitor data source change events: the data synchronization unit monitors data source change events, such as detecting changes in weather forecast data at a meteorological station, and executing a multi-dimensional incremental synchronization algorithm: using an intelligent incremental synchronization algorithm, the algorithm comprehensively considers multi-dimensional factors such as historical change records, data source characteristics, data item importance, data source quality score, and data source credibility to predict the possibility of data item changes and record version numbers and timestamps: during the synchronization process, the data synchronization unit records the version number and timestamp of each data item as synchronization information for use by the conflict resolution unit.

[0150] Beneficial effects: Through the multi-dimensional incremental synchronization algorithm, changes in data sources can be captured in a timely manner, and data in the virtualization system can be quickly updated to ensure data consistency and accuracy.

[0151] During the synchronization process, the data synchronization unit records the version number and timestamp of each data item as synchronization information for use by the conflict resolution unit;

[0152] In this step, it is checked whether there is a data conflict. If a data conflict is detected, a consistency assurance algorithm is used to resolve the conflict. The conflict resolution unit receives the synchronization information transmitted by the data synchronization unit, including the version number and timestamp of each data item. The conflict resolution unit checks whether there is a data conflict. For example, it is detected that two data sources have made different updates to the same data item. When a data conflict is detected, the conflict resolution unit uses a consistency assurance algorithm to resolve the conflict. The algorithm resolves data conflicts through data version control, timestamp mechanism, priority strategy and other means to ensure data consistency and accuracy.

[0153] Beneficial effects: Through the consistency guarantee algorithm, the data conflict problem is solved, the consistency and accuracy of the data are ensured, and the availability and reliability of the data are improved.

[0154] The conflict resolution unit returns the resolved data version to the data synchronization unit, and the data synchronization unit updates the data in the virtualization system accordingly;

[0155] The data synchronization unit and the conflict resolution unit communicate with each other through a message passing mechanism to ensure the coordination and consistency of the synchronization process and the conflict resolution process.

[0156] In an embodiment of the present invention, the data synchronization unit and the conflict resolution unit communicate through a message passing mechanism to ensure the coordination and consistency of the synchronization process and the conflict resolution process. After monitoring the data source change event, the data synchronization unit first executes a multi-dimensional incremental synchronization algorithm to synchronize the changed data to the virtualization system. During the synchronization process, the data synchronization unit records the version number and timestamp of each data item as synchronization information. After the synchronization is completed, the data synchronization unit passes the synchronization information to the conflict resolution unit. After receiving the synchronization information, the conflict resolution unit checks whether there is a data conflict. If a data conflict is detected, the conflict resolution unit uses a consistency assurance algorithm to resolve the conflict. The conflict resolution unit returns the resolved data version to the data synchronization unit, and the data synchronization unit updates the data in the virtualization system accordingly.

[0157] Beneficial effects: Through the coordination and consistency between the data synchronization unit and the conflict resolution unit, the efficiency and accuracy of data synchronization are ensured, and the stability and reliability of the system are improved.

[0158] Furthermore, the multi-dimensional incremental synchronization algorithm includes:

[0159] Given a data item D i Historical change records of H i , data source S j Characteristics of C j , the importance of data items W i , the quality score Q of the data source j , and the data source credibility T j , predict data item D i The predicted probability P of a change occurring in the next time period i for:

[0160] ;

[0161] in, is the sigmoid function, which is used to convert the prediction results into probability values. Indicates extracting historical records H through time series model i The characteristic vector of Represents the data source characteristics C obtained through cluster analysis j The characteristic vector of Represents the importance of the data item predicted by the prediction model W i The characteristic vector of Indicates the quality score Q of the data source j The nonlinear weight function, Indicates the credibility of the data source T j The characteristic vector of , , , , is the weight coefficient, which is used to adjust the influence of each factor;

[0162] For data items with a predicted probability greater than a threshold, data synchronization is performed in the previous time period of the preset time node. For data items with a predicted probability less than a threshold, data synchronization is performed in the next time period of the preset time node to reduce data redundancy.

[0163] In an embodiment of the present invention, a machine learning model is first used to predict the possibility of change of a data item, and the synchronization strategy is dynamically adjusted according to the prediction result. The historical change records, data source characteristics, data item importance, data source quality score and other dimensions are comprehensively considered. The weight in data fusion is dynamically adjusted according to the quality score of the data source, a nonlinear function is used to amplify the weight of high-quality data sources, a time series model is used to predict the possibility of future changes, the credibility of the data source is evaluated through a machine learning model, and finally the prediction threshold is dynamically adjusted according to the characteristics of the data source.

[0164] First of all, in the data preparation work, we collect the historical change records of data items, then obtain the characteristics of the data source, evaluate the importance of the data items, calculate the quality score of the data source, and evaluate the credibility of the data source.

[0165] Then feature extraction is performed. Time series models (such as ARIMA, Prophet, etc.) are used to extract feature vectors of historical change records. Cluster analysis is used to obtain feature vectors of data source characteristics. Feature vectors of data item importance are predicted through expert evaluation or machine learning models. A nonlinear weight function is used to calculate the quality score of the data source using a nonlinear function. Feature vectors of data source credibility are predicted through machine learning models.

[0166] Secondly, the probability is predicted: based on the feature vector of historical change records, the feature vector of data source characteristics, the feature vector of data item importance, the nonlinear weight function of data source quality score, and the feature vector of data source credibility, the sigmoid function is used to calculate the probability of data item change.

[0167] Finally, dynamically adjust the synchronization strategy according to the predicted probability. For data items with high predicted probability, synchronize in advance to reduce delays. For data items with low predicted probability, delay synchronization appropriately to reduce redundancy. Adjust the synchronization priority of data items according to the quality score of the data source and the nonlinear weight function. Adjust the synchronization frequency of data items according to the credibility of the data source.

[0168] In this way, the intelligent incremental synchronization algorithm can consider multiple factors more comprehensively and predict the possibility of change of data items through complex mathematical models, thereby optimizing the incremental synchronization strategy and improving the efficiency and accuracy of data synchronization.

[0169] In order to clarify the time node of "advance" or "delay", it is necessary to define a reference time point or period to determine when to synchronize data. This reference time point or period can be flexibly set according to the actual situation, but it can usually be determined based on the following aspects:

[0170] Regular synchronization cycle: Set a fixed synchronization cycle, such as once an hour or once a day. During this cycle, the specific time of advance or delay is determined based on the predicted probability of the data item.

[0171] Data source update frequency: Set the benchmark time point according to the data source update frequency. For example, if the data source is updated once an hour, you can set each hour as a cycle.

[0172] Importance of data items: For very important data items, even if the prediction probability is not high, it may be necessary to set a shorter synchronization period to ensure the timeliness of the data.

[0173] Assume that a regular synchronization cycle is set to once an hour. This means that all data items will be synchronized at a fixed time point every hour by default. This time point can be adjusted based on the predicted probability:

[0174] For data items with high predicted probability: If the predicted probability is higher than a certain threshold (for example, 0.7), you can synchronize in advance, such as 15 minutes before the scheduled synchronization time, to ensure that data items with high probability of change are updated in time and reduce delays.

[0175] For data items with low predicted probabilities: If the predicted probability is lower than a certain threshold (for example, 0.3), synchronization can be delayed, for example, 15 minutes after the scheduled synchronization time. This can reduce unnecessary synchronization times and reduce redundancy.

[0176] If the data source is updated once an hour, the synchronization time can be adjusted based on the predicted probability: For data items with high predicted probability: If the predicted probability is higher than a certain threshold, synchronization can be performed immediately after the data source is updated, or within a few minutes after the data source is updated to ensure the timeliness of the data.

[0177] For data items with low prediction probability: If the prediction probability is lower than a certain threshold, synchronization can be performed at a later time after the data source is updated, for example, half an hour or more after the data source is updated, to reduce unnecessary synchronization operations.

[0178] For some very important data items, even if the prediction probability is not high, you may need to set a shorter synchronization period to ensure the timeliness of the data:

[0179] For important data items, regardless of the predicted probability, a shorter synchronization cycle can be set, such as synchronization every half hour or every 15 minutes, to ensure that these data items are updated in a timely manner.

[0180] Example:

[0181] Assume that a regular synchronization cycle is set to once an hour, the data source is updated once an hour, and the importance of the data item has been evaluated. The time node for early or delayed synchronization can be defined as follows:

[0182] For data items with high prediction probability (for example, prediction probability greater than 0.7), synchronization is performed immediately after the data source is updated, or within 5 minutes after the data source is updated. For data items with low prediction probability (for example, prediction probability less than 0.3), synchronization is performed within 30 minutes after the data source is updated. For important data items (regardless of the prediction probability), a shorter synchronization cycle is set, such as synchronization every half an hour. Through such time node definition, the timeliness and efficiency of data synchronization can be effectively balanced, ensuring data consistency and accuracy, while reducing unnecessary synchronization operations.

[0183] A specific embodiment of a data mapping layer module in an emergency data standardization system based on data virtualization includes a detailed explanation of a type determination unit, a virtual table construction unit, an execution unit, a query unit, and specific steps of data conversion using Box-Cox transformation and an analysis of a specific embodiment. Taking the field of natural disaster emergency response as an example, in another embodiment provided by the present invention, the data mapping layer module includes:

[0184] A type determination unit, used to determine the types of each data source in the metadata extraction module, and determine a packaging table based on the data source type to parse the emergency data;

[0185] Determine the types of data sources in the metadata extraction module, and determine the encapsulation table based on the data source type to parse the emergency data.

[0186] Example:

[0187] The type determination unit identifies the type of each data source in the metadata extraction module, such as a weather station, an earthquake monitoring station, a river water level monitoring station, etc., and determines a packaging table based on the data source type. For example, for the data from a weather station, a packaging table is determined to parse the weather forecast data.

[0188] A virtual table construction unit, used to define a virtual table based on a preset encapsulation table, wherein the virtual table is determined according to the logical structure of the emergency data;

[0189] A virtual table is defined based on a preset encapsulation table, wherein the virtual table is determined according to the logical structure of the emergency data.

[0190] Example:

[0191] The virtual table construction unit defines a virtual table according to a preset encapsulated table, for example, a virtual table is defined to represent weather forecast data, including fields such as timestamp, temperature, humidity, etc., and Box-Cox transformation is used to convert data to improve the normality of data distribution.

[0192] An execution unit, used for executing mapping rules to convert the emergency data in the data source into a format of a virtual table;

[0193] Execute mapping rules to convert the emergency data in the data source into the format of the virtual table.

[0194] Example:

[0195] The execution unit converts the emergency data in the data source into a virtual table format according to the mapping rule, for example, converts the weather forecast data of the meteorological station into a predefined virtual table format.

[0196] The query unit is used to establish a query port to enable upper layer data applications to access emergency data.

[0197] The embodiment of the present invention establishes a query port by establishing a query unit, so that an upper layer data application can access the emergency data in the virtual table through a unified interface.

[0198] Wherein, when emergency data conversion is involved in the virtual table construction unit in the data mapping layer module, Box-Cox transformation is used to perform data conversion to improve the normality of data distribution, wherein the step of using Box-Cox transformation to perform data conversion to improve the normality of data distribution includes:

[0199] The maximum likelihood estimation method is used to determine the optimal λ value, where the λ value is used to determine the type of transformation;

[0200] For each non-negative emergency data value, if the λ value is not equal to zero, the power transformation method is used to calculate the λ power of the original data value, subtract one from the result of the λ power, and divide the result by the λ value;

[0201] If the λ value is equal to zero, the logarithmic transformation method is used to take the natural logarithm of the original data value;

[0202] After applying the transformation, normality test methods were used to verify whether the transformed data were closer to a normal distribution;

[0203] If the transformed data verified normality, standardization was performed.

[0204] The embodiment of the present invention improves the normality of data distribution by defining a virtual table and adopting Box-Cox transformation, thereby improving the availability and applicability of data, facilitating subsequent data analysis and decision support, and by establishing a query port, allowing upper-level data applications to access emergency data in the virtual table through a unified interface, thereby improving the convenience of data access, and by executing mapping rules, ensuring the consistency and accuracy of data, improving the reliability and effectiveness of emergency response decisions, and by determining the data source type and defining the corresponding virtual table, improving the flexibility and scalability of the system, and being able to adapt to data of different types and sources.

[0205] In another embodiment of the present invention, a method for standardizing emergency data based on data virtualization is provided. Figure 2 As shown, including:

[0206] A control metadata extraction module extracts metadata through multiple different data source interfaces and sends the metadata to a virtualization system to serve as a basis for defining data views and virtual tables, wherein the metadata includes multiple types of heterogeneous emergency data;

[0207] Controlling the heterogeneous data fusion module to receive the metadata of the metadata extraction module, performing data fusion through the data virtualization server, and determining fused data so that upper layer applications can access multiple types of heterogeneous data;

[0208] The control data standardization module standardizes the fused data based on the preset emergency data standard to obtain emergency data;

[0209] The control mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; when multiple data sources are detected to modify the same data, the consistency and accuracy of the data is ensured through a consistency assurance algorithm;

[0210] The control data mapping layer module maps the emergency data obtained after standardization to the original data source according to the mapping of the virtual table to the data source.

[0211] An embodiment of the present invention further provides an electronic device, including:

[0212] processor;

[0213] a memory for storing processor-executable instructions;

[0214] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0215] According to a fourth aspect of the embodiments of the present invention,

[0216] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0217] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An emergency data standardization system based on data virtualization, characterized in that: include: Metadata extraction module, heterogeneous data fusion module, data standardization module, mapping pre-processing module and data mapping layer module, among which: The metadata extraction module extracts metadata through multiple different data source interfaces and sends the metadata to the virtualization system as a basis for data view and virtual table definition, wherein the metadata includes multiple types of heterogeneous emergency data; The heterogeneous data fusion module receives the metadata of the metadata extraction module, performs data fusion through the data virtualization server, and determines the fused data so that the upper layer application can access multiple types of heterogeneous data; The data standardization module standardizes the fused data based on a preset emergency data standard to obtain emergency data; The mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; when multiple data sources are detected to modify the same data, the consistency and accuracy of the data is ensured by a consistency assurance algorithm; The data mapping layer module maps the emergency data obtained after the standardization process to the original data source according to the mapping of the virtual table to the data source; Wherein, the mapping pre-processing module includes: a data synchronization unit and a conflict resolution unit; After detecting a data source change event, the data synchronization unit first executes a multi-dimensional incremental synchronization algorithm to synchronize the changed data to the virtualization system; During the synchronization process, the data synchronization unit records the version number and timestamp of each data item as synchronization information for use by the conflict resolution unit; After the synchronization is completed, the data synchronization unit transmits the synchronization information to the conflict resolution unit; After receiving the synchronization information, the conflict resolution unit checks whether there is a data conflict; If a data conflict is detected, the conflict resolution unit uses a consistency guarantee algorithm to resolve the conflict; The conflict resolution unit returns the resolved data version to the data synchronization unit, and the data synchronization unit updates the data in the virtualization system accordingly; The data synchronization unit and the conflict resolution unit communicate with each other through a message passing mechanism to ensure the coordination and consistency of the synchronization process and the conflict resolution process; The multi-dimensional incremental synchronization algorithm includes: Given a data item D i Historical change records of H i , data source S j Characteristics of C j , the importance of data items W i , the quality score Q of the data source j , and the data source credibility T j , predict data item D i The predicted probability P of a change occurring in the next time period i for: ; in, is the sigmoid function, which is used to convert the prediction results into probability values. Indicates extracting historical records H through time series model i The characteristic vector of Represents the data source characteristics C obtained through cluster analysis j The characteristic vector of Represents the importance of the data item predicted by the prediction model W i The characteristic vector of Indicates the quality score Q of the data source j The nonlinear weight function, Indicates the credibility of the data source T j The characteristic vector of , , , , is the weight coefficient, which is used to adjust the influence of each factor; For data items with a predicted probability greater than a threshold, data synchronization is performed in the previous time period of the preset time node. For data items with a predicted probability less than a threshold, data synchronization is performed in the next time period of the preset time node to reduce data redundancy.

2. The emergency data standardization system based on data virtualization according to claim 1 is characterized in that: The heterogeneous data fusion module includes: A data source identification unit, used to identify the type of each data source connected to the data virtualization server; The data adapter unit configures a data adapter for each type of data source to parse the structure and content of the data source to convert metadata in different formats into a unified intermediate representation; A data mapping unit analyzes the type of metadata to establish a mapping relationship between the data source and the internal data model of the virtualization system, so that the metadata can be converted between different data sources and the virtualization system; The data fusion unit receives the intermediate representation data converted by the data adapter unit and executes the data fusion algorithm to create a data view so that the upper-layer application can access the data from different data sources with a unified interface.

3. The emergency data standardization system based on data virtualization according to claim 2 is characterized in that: The data fusion unit comprises: Obtaining the number of records of the data source in the metadata extraction module to determine the integrity of the data source; Determining the consistency of the data source based on the mapping relationship between the data source established by the data mapping unit and the internal data model of the virtualization system; Determining the timeliness of data source update based on data source update events in the metadata extraction module; Determining the accuracy of the data source based on the emergency data and metadata outputted by the data standardization module; Obtaining the data source and preset measurement indicators in the metadata extraction module to determine the reliability of the data source; Based on the data mapping layer module, the standardized emergency data is mapped to the original data source to determine the richness of the data source; Calculate the overall quality score of the data source based on the consistency, timeliness, accuracy, reliability and richness of the data source; When the comprehensive quality score is greater than a preset value, an adaptive data fusion strategy is used to construct a data view.

4. The emergency data standardization system based on data virtualization according to claim 1 is characterized in that: The data mapping layer module includes: A type determination unit, used to determine the types of each data source in the metadata extraction module, and determine a packaging table based on the data source type to parse the emergency data; A virtual table construction unit, used to define a virtual table based on a preset encapsulation table, wherein the virtual table is determined according to the logical structure of the emergency data; An execution unit, used for executing mapping rules to convert the emergency data in the data source into a format of a virtual table; The query unit is used to establish a query port to enable upper layer data applications to access emergency data.

5. The emergency data standardization system based on data virtualization according to claim 4 is characterized in that: When emergency data conversion is involved in the virtual table construction unit in the data mapping layer module, Box-Cox transformation is used to perform data conversion to improve the normality of data distribution, wherein the step of using Box-Cox transformation to perform data conversion to improve the normality of data distribution includes: The maximum likelihood estimation method is used to determine the optimal λ value, where the λ value is used to determine the type of transformation; For each non-negative emergency data value, if the λ value is not equal to zero, the power transformation method is used to calculate the λ power of the original data value, subtract one from the result of the λ power, and divide the result by the λ value; If the λ value is equal to zero, the logarithmic transformation method is used to take the natural logarithm of the original data value; After applying the transformation, normality test methods were used to verify whether the transformed data were closer to a normal distribution; If the transformed data verified normality, standardization was performed.

6. A method for emergency data standardization based on data virtualization, characterized in that: include: A control metadata extraction module extracts metadata through multiple different data source interfaces and sends the metadata to a virtualization system to serve as a basis for defining data views and virtual tables, wherein the metadata includes multiple types of heterogeneous emergency data; Controlling the heterogeneous data fusion module to receive the metadata of the metadata extraction module, performing data fusion through the data virtualization server, and determining fused data so that upper layer applications can access multiple types of heterogeneous data; The control data standardization module standardizes the fused data based on the preset emergency data standard to obtain emergency data; The control mapping pre-processing module obtains the emergency data format based on metadata analysis, and then monitors the data source according to the emergency data format. When a change event of the data source is detected, a multi-dimensional incremental synchronization algorithm is used to ensure that the emergency data in the virtualization system is consistent with the data source; When multiple data sources are detected to modify the same data, the consistency and accuracy of the data is ensured through the consistency assurance algorithm; The control data mapping layer module maps the emergency data obtained after standardization to the original data source according to the mapping of the virtual table to the data source; Wherein, the mapping pre-processing module includes: a data synchronization unit and a conflict resolution unit; After detecting a data source change event, the data synchronization unit first executes a multi-dimensional incremental synchronization algorithm to synchronize the changed data to the virtualization system; During the synchronization process, the data synchronization unit records the version number and timestamp of each data item as synchronization information for use by the conflict resolution unit; After the synchronization is completed, the data synchronization unit transmits the synchronization information to the conflict resolution unit; After receiving the synchronization information, the conflict resolution unit checks whether there is a data conflict; If a data conflict is detected, the conflict resolution unit uses a consistency guarantee algorithm to resolve the conflict; The conflict resolution unit returns the resolved data version to the data synchronization unit, and the data synchronization unit updates the data in the virtualization system accordingly; The data synchronization unit and the conflict resolution unit communicate with each other through a message passing mechanism to ensure the coordination and consistency of the synchronization process and the conflict resolution process; The multi-dimensional incremental synchronization algorithm includes: Given a data item D i Historical change records of H i , data source S j Characteristics of C j , the importance of data items W i , the quality score Q of the data source j , and the data source credibility T j , predict data item D i The predicted probability P of a change occurring in the next time period i for: ; in, is the sigmoid function, which is used to convert the prediction results into probability values. Indicates extracting historical records H through time series model i The characteristic vector of Represents the data source characteristics C obtained through cluster analysis j The characteristic vector of Represents the importance of the data item predicted by the prediction model W i The characteristic vector of Indicates the quality score Q of the data source j The nonlinear weight function, Indicates the credibility of the data source T j The characteristic vector of , , , , is the weight coefficient, which is used to adjust the influence of each factor; For data items with a predicted probability greater than a threshold, data synchronization is performed in the previous time period of the preset time node. For data items with a predicted probability less than a threshold, data synchronization is performed in the next time period of the preset time node to reduce data redundancy.

7. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the emergency data standardization method based on data virtualization as described in claim 6.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, wherein the process includes the emergency data standardization method based on data virtualization according to claim 6.

Citation Information

Patent Citations

  • Power grid disaster early warning method and system based on multi-source heterogeneous data fusion model

    CN116739336A

  • Order synchronization method of business management platform

    CN117725122A

  • Data analysis processing system based on big data

    CN117851490A

  • Cross-platform digital textbook synchronization method and system

    CN117931823A

  • Intelligent data synchronization method for heterogeneous database

    CN118035358A