Multivariate data processing method and system

Abstract

The embodiment of the invention provides a multivariate data processing method and system.The method comprises the steps that original data are collected from multiple data sources by calling multiple APIs, and multi-source heterogeneous data are obtained; formatting the multi-source heterogeneous data according to a preset format conversion rule, wherein the formatting comprises timestamp alignment, space coordinate system conversion and key frame extraction; performing multi-dimensional feature extraction on the unified data to generate space-time aligned feature vectors, and performing cross-modal data fusion on the feature vectors based on a dynamic weight distribution algorithm to generate a fusion data packet; and performing data quality evaluation on the fused data packet, including verifying the integrity, consistency and real-time performance of the data, and dynamically adjusting a transmission path of a data acquisition interface according to a verification result. According to the method, an efficient and reliable solution is provided for collection, processing and fusion of multi-source heterogeneous data, and the data processing efficiency and the decision support capability are remarkably improved.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a multivariate data processing method and system. Background Art

[0002] With the rapid development of the Industrial Internet of Things, intelligent monitoring, and spatial perception technologies, multi-dimensional data processing technology has become a core support for smart manufacturing, smart cities, and other fields. Traditional data processing systems need to integrate heterogeneous data from sensors, audio and video equipment, geographic information systems, and databases, but they face the following technical bottlenecks in practical applications:

[0003] Inefficient multi-source data collaboration: Existing technologies often rely on fixed rules to process heterogeneous data, such as unified timestamp alignment or static coordinate system conversion. These methods struggle to adapt to dynamic changes in industrial scenarios, such as differences in device sampling rates and network fluctuations. This leads to accumulated data alignment errors. Traditional keyframe extraction strategies are particularly prone to losing valuable information when processing audio and video streams, and can exacerbate transmission delays when bandwidth is limited.

[0004] Rigid data fusion strategies: Existing cross-modal fusion methods often use static weight allocation, failing to consider the dynamic changes in device credibility and differences in data real-time requirements, resulting in fusion results that deviate from real-world scenarios.

[0005] Quality assessment is disconnected from resource optimization: Traditional data verification is limited to integrity checks and lacks in-depth analysis of data consistency and real-time offsets. Furthermore, transmission path adjustments rely on manual configuration and cannot dynamically switch between local preprocessing and cloud computing based on network status, resulting in wasted resources.

[0006] Unexplainable decision-making process: While deep learning-based fusion models improve analytical accuracy, their "black box" nature results in a lack of traceability of compliance testing results, making it difficult to meet the audit evidence chain requirements of highly regulated industries such as finance and healthcare.

[0007] Insufficient edge resource utilization: Existing architectures mostly use centralized cloud computing, which results in a large amount of bandwidth being consumed when transmitting audio and video data to the cloud for processing. In addition, edge nodes only perform simple collection functions and do not fully utilize their computing power for real-time preprocessing.

[0008] To address these issues, one patent proposes matching video keyframes with audio keywords to improve behavior detection accuracy. However, its cross-modal association relies on fixed rules and cannot handle complex spatiotemporal semantics. Another patent proposes a framing strategy to optimize video analysis efficiency, but fails to incorporate a dynamic frame extraction mechanism that adapts to content complexity, resulting in low-information frames still occupying bandwidth. Furthermore, existing solutions generally neglect the integration of device credibility assessment with federated learning frameworks, making it difficult to achieve multi-node collaborative optimization. Summary of the Invention

[0009] One or more embodiments of this specification provide a multivariate data processing method, including:

[0010] By calling multiple API interfaces, raw data is collected from multiple data sources to obtain multi-source heterogeneous data;

[0011] Formatting the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion, and key frame extraction;

[0012] Perform multi-dimensional feature extraction on the unified data to generate spatiotemporally aligned feature vectors, and perform cross-modal data fusion on the feature vectors based on a dynamic weight allocation algorithm to generate a fused data packet;

[0013] Perform data quality assessment on the fused data packet, including verification of data integrity, consistency and real-time performance, and dynamically adjust the transmission path of the data acquisition interface based on the verification results.

[0014] Furthermore, when calling multiple API interfaces to collect raw data from multiple data sources, it also includes:

[0015] Through the intelligent caching mechanism, data from data sources that have been frequently collected recently are cached. When collecting data from the same data source next time, it is obtained from the cache first. If the data in the cache is expired or does not exist, it is collected from the data source again.

[0016] Furthermore, the preset format conversion rules include:

[0017] The time stamps of multi-source heterogeneous data are calibrated through the time synchronization protocol to ensure that the timestamps of different data sources are aligned under the same time base, and the data with missing timestamps is supplemented by the interpolation algorithm;

[0018] The spatial coordinate systems of different data sources are converted into the target coordinate system through the Geographic Information System (GIS), and the spatial data are projected and transformed using the coordinate transformation algorithm to ensure that the spatial data are aligned in the same coordinate system.

[0019] For audio and video streaming data, key frames of the video data are detected and extracted through image processing algorithms, and compressed sensing technology is used to compress the key frames.

[0020] Furthermore, the multi-dimensional feature extraction of the unified data is specifically as follows:

[0021] A feature dimensionality reduction method based on the combination of principal component analysis (PCA) and linear discriminant analysis (LDA) is used:

[0022] Use PCA to perform linear transformation on high-dimensional data, determine the number of principal components based on the cumulative contribution rate threshold, and remove noise and redundant information;

[0023] The data after PCA dimensionality reduction is then input into the LDA model to optimize the feature projection direction by maximizing the ratio of the inter-class scatter matrix and minimizing the intra-class scatter matrix.

[0024] Furthermore, the weight distribution of the dynamic weight distribution algorithm is dynamically adjusted based on the real-time index of the data source, the device credibility score and the historical error feedback, specifically:

[0025]

[0026] Among them, T i is the data real-time coefficient, T max is the maximum delay threshold allowed by the system, C i is the credibility coefficient based on the historical behavior evaluation of the device, E i is the historical fusion error value, E base is the benchmark error value, α, β, γ are configurable adjustment parameters.

[0027] Furthermore, the data quality assessment of the fused data packet specifically includes:

[0028] The integrity of the data is verified by detecting missing data fields. If a field is detected to be missing, a retransmission request of the corresponding interface is triggered;

[0029] Verify data consistency by comparing the attribute description difference threshold of the same entity in multi-source data;

[0030] A correction instruction is generated based on the offset between the data generation timestamp and the system clock to verify the real-time nature of the data.

[0031] Furthermore, the specific method for dynamically adjusting the data acquisition interface transmission path according to the verification results is as follows:

[0032] Build a reinforcement learning model based on deep Q network DQN;

[0033] Taking historical verification results, transmission path usage, and current network status as state inputs, we design a reward function that gives positive rewards to stable and efficient transmission paths and negative rewards to paths that cause data quality issues.

[0034] The network parameters are updated through the Q learning algorithm, and the experience replay mechanism is used to break the data correlation to achieve intelligent selection of the optimal data acquisition interface transmission path.

[0035] One or more embodiments of this specification provide a multivariate data processing system, including:

[0036] Data acquisition module: used to collect raw data from multiple data sources by calling multiple API interfaces to obtain multi-source heterogeneous data;

[0037] Data preprocessing module: used to format the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion and key frame extraction;

[0038] Data fusion module: used to extract multi-dimensional features from the unified data, generate spatiotemporally aligned feature vectors, perform cross-modal data fusion on the feature vectors based on a dynamic weight allocation algorithm, and generate a fused data packet;

[0039] Data optimization module: used to evaluate the data quality of the fused data packet, including data integrity, consistency and real-time verification, and dynamically adjust the transmission path of the data acquisition interface based on the verification results.

[0040] One or more embodiments of this specification provide an electronic device, including:

[0041] processor; and,

[0042] A memory is arranged to store computer executable instructions, which, when executed, cause the processor to implement the steps of the multivariate data processing method described above.

[0043] One or more embodiments of this specification provide a storage medium for storing computer-executable instructions, which implement the steps of the multivariate data processing method when executed.

[0044] By adopting the embodiment of the present invention, raw data is collected from multi-source heterogeneous data sources by calling multiple API interfaces, which solves the problems of single data source and low collaborative efficiency of traditional methods; multi-source heterogeneous data is formatted according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion and key frame extraction, which significantly improves the consistency and availability of data; by extracting multi-dimensional features from the unified data, generating spatiotemporal aligned feature vectors, and combining cross-modal data fusion with a dynamic weight allocation algorithm, a high-precision fusion data packet is generated, which effectively improves the accuracy and adaptability of data fusion; in addition, by checking the integrity, consistency and real-time performance of the fusion data packet, and dynamically adjusting the transmission path of the data acquisition interface according to the verification results, the high quality and real-time performance of the data are ensured, while optimizing resource utilization. The present invention provides an efficient and reliable solution for the collection, processing and fusion of multi-source heterogeneous data, which is suitable for fields such as intelligent manufacturing and smart cities, and significantly improves data processing efficiency and decision support capabilities.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of a multivariate data processing method provided for one or more embodiments of this specification;

[0048] Figure 2 A schematic diagram of the composition of a multivariate data processing system provided for one or more embodiments of this specification;

[0049] Figure 3 A schematic diagram of the structure of an electronic device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0051] Method Example

[0052] According to an embodiment of the present invention, a multivariate data processing method is provided. Figure 1 A flowchart of a multivariate data processing method provided in one or more embodiments of this specification is shown in FIG. Figure 1 As shown, the multivariate data processing method according to an embodiment of the present invention specifically includes:

[0053] S1. Collect raw data from multiple data sources by calling multiple API interfaces to obtain multi-source heterogeneous data.

[0054] First, identify the various data sources you need to collect, such as databases (including relational and non-relational databases); file data (including CSV and JSON files); various network platforms, the Industrial Internet of Things, spatial data (including images, maps, models, point cloud data, audio, video, and image data), etc. For each data source, determine the API interface it provides, including the interface address, request method, request parameter format, and authentication method. For multiple data source APIs, call different API interfaces sequentially or simultaneously to collect data from each source.

[0055] When calling multiple API interfaces to collect raw data from multiple data sources, it also includes:

[0056] Through the intelligent caching mechanism, data from data sources that have been frequently collected recently are cached. When collecting data from the same data source next time, it is obtained from the cache first. If the data in the cache is expired or does not exist, it is collected from the data source again.

[0057] S2. Formatting the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion, and key frame extraction.

[0058] The preset format conversion rules include:

[0059] The time stamps of multi-source heterogeneous data are calibrated through the time synchronization protocol to ensure that the timestamps of different data sources are aligned under the same time base, and the data with missing timestamps is supplemented by the interpolation algorithm;

[0060] The spatial coordinate systems of different data sources are converted into the target coordinate system through the Geographic Information System (GIS), and the spatial data are projected and transformed using the coordinate transformation algorithm to ensure that the spatial data are aligned in the same coordinate system.

[0061] For audio and video streaming data, key frames of the video data are detected and extracted through image processing algorithms, and compressed sensing technology is used to compress the key frames.

[0062] S3. Perform multi-dimensional feature extraction on the unified data to generate a spatiotemporally aligned feature vector. Based on a dynamic weight allocation algorithm, perform cross-modal data fusion on the feature vector to generate a fused data packet.

[0063] The specific steps for multi-dimensional feature extraction of unified data are as follows:

[0064] A feature dimensionality reduction method based on the combination of principal component analysis (PCA) and linear discriminant analysis (LDA) is used:

[0065] Use PCA to perform linear transformation on high-dimensional data, determine the number of principal components based on the cumulative contribution rate threshold, and remove noise and redundant information;

[0066] The data after PCA dimensionality reduction is then input into the LDA model to optimize the feature projection direction by maximizing the ratio of the inter-class scatter matrix and minimizing the intra-class scatter matrix.

[0067] The weight distribution of the dynamic weight distribution algorithm is dynamically adjusted based on the real-time indicators of the data source, the device credibility score and historical error feedback. Specifically:

[0068]

[0069] Among them, T i is the data real-time coefficient, T max is the maximum delay threshold allowed by the system, C i is the credibility coefficient based on the historical behavior evaluation of the device, E i is the historical fusion error value, E base is the benchmark error value, α, β, γ are configurable adjustment parameters.

[0070] S4. Perform data quality assessment on the fused data packet, including verification of data integrity, consistency, and real-time performance, and dynamically adjust the transmission path of the data acquisition interface based on the verification results.

[0071] The data quality assessment of the fused data package specifically includes:

[0072] The integrity of the data is verified by detecting missing data fields. If a field is detected to be missing, a retransmission request of the corresponding interface is triggered;

[0073] Verify data consistency by comparing the attribute description difference threshold of the same entity in multi-source data;

[0074] A correction instruction is generated based on the offset between the data generation timestamp and the system clock to verify the real-time nature of the data.

[0075] The specific method for dynamically adjusting the data acquisition interface transmission path based on the verification results is as follows:

[0076] Build a reinforcement learning model based on deep Q network DQN;

[0077] Taking historical verification results, transmission path usage, and current network status as state inputs, we design a reward function that gives positive rewards to stable and efficient transmission paths and negative rewards to paths that cause data quality issues.

[0078] The network parameters are updated through the Q learning algorithm, and the experience replay mechanism is used to break the data correlation to achieve intelligent selection of the optimal data acquisition interface transmission path.

[0079] The beneficial effects of the present invention are as follows:

[0080] The present invention collects raw data from multi-source heterogeneous data sources by calling multiple API interfaces, solving the problems of single data source and low collaborative efficiency of traditional methods; formats the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion and key frame extraction, significantly improving the consistency and availability of the data; extracts multi-dimensional features from the unified data, generates spatiotemporal aligned feature vectors, and combines cross-modal data fusion with a dynamic weight allocation algorithm to generate high-precision fused data packets, effectively improving the accuracy and adaptability of data fusion; in addition, by checking the integrity, consistency and real-time performance of the fused data packets, and dynamically adjusting the transmission path of the data acquisition interface according to the verification results, the high quality and real-time performance of the data are ensured, while optimizing resource utilization. The present invention provides an efficient and reliable solution for the collection, processing and fusion of multi-source heterogeneous data, which is suitable for fields such as intelligent manufacturing and smart cities, and significantly improves data processing efficiency and decision-making support capabilities.

[0081] System Example

[0082] According to an embodiment of the present invention, a multivariate data processing system is provided. Figure 2 A schematic diagram of a multivariate data processing system provided in one or more embodiments of this specification is shown in FIG. Figure 2 As shown, the multivariate data processing system according to an embodiment of the present invention specifically includes:

[0083] Data collection module 20: used to collect raw data from multiple data sources by calling multiple API interfaces to obtain multi-source heterogeneous data;

[0084] Data preprocessing module 22: used to format the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion and key frame extraction;

[0085] Data fusion module 24: used to extract multi-dimensional features from the unified data, generate spatiotemporally aligned feature vectors, perform cross-modal data fusion on the feature vectors based on a dynamic weight allocation algorithm, and generate a fused data packet;

[0086] Data optimization module 26: used to evaluate the data quality of the fused data packet, including data integrity, consistency and real-time verification, and dynamically adjust the transmission path of the data acquisition interface according to the verification results.

[0087] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0088] Device Example 1

[0089] An embodiment of the present invention provides an electronic device, such as Figure 3 As shown, it includes: a memory 30, a processor 32, and a computer program stored in the memory 30 and executable on the processor 32. When the computer program is executed by the processor 32, the following method steps are implemented:

[0090] S1. Collect raw data from multiple data sources by calling multiple API interfaces to obtain multi-source heterogeneous data;

[0091] S2. Formatting the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion, and key frame extraction;

[0092] S3. Perform multi-dimensional feature extraction on the unified data to generate spatiotemporally aligned feature vectors, and perform cross-modal data fusion on the feature vectors based on a dynamic weight allocation algorithm to generate a fused data packet;

[0093] S4. Perform data quality assessment on the fused data packet, including verification of data integrity, consistency, and real-time performance, and dynamically adjust the transmission path of the data acquisition interface based on the verification results.

[0094] Device Example 2

[0095] An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program for implementing information transmission. When the program is executed by the processor 32, the following method steps are implemented:

[0096] S1. Collect raw data from multiple data sources by calling multiple API interfaces to obtain multi-source heterogeneous data;

[0097] S2. Formatting the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion, and key frame extraction;

[0098] S3. Perform multi-dimensional feature extraction on the unified data to generate spatiotemporally aligned feature vectors, and perform cross-modal data fusion on the feature vectors based on a dynamic weight allocation algorithm to generate a fused data packet;

[0099] S4. Perform data quality assessment on the fused data packet, including verification of data integrity, consistency, and real-time performance, and dynamically adjust the transmission path of the data acquisition interface based on the verification results.

[0100] The computer-readable storage medium in this embodiment includes but is not limited to: ROM, RAM, magnetic disk or optical disk, etc.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multivariate data processing method, characterized in that: include: By calling multiple API interfaces, raw data is collected from multiple data sources to obtain multi-source heterogeneous data; Formatting the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion, and key frame extraction; Perform multi-dimensional feature extraction on the unified data to generate spatiotemporally aligned feature vectors, and perform cross-modal data fusion on the feature vectors based on a dynamic weight allocation algorithm to generate a fused data packet; Perform data quality assessment on the fused data packet, including verification of data integrity, consistency and real-time performance, and dynamically adjust the transmission path of the data acquisition interface based on the verification results.

2. The method according to claim 1, characterized in that When calling multiple API interfaces to collect raw data from multiple data sources, it also includes: Through the intelligent caching mechanism, data from data sources that have been frequently collected recently are cached. When collecting data from the same data source next time, it is obtained from the cache first. If the data in the cache is expired or does not exist, it is collected from the data source again.

3. The method according to claim 1, characterized in that The preset format conversion rules include: The time stamps of multi-source heterogeneous data are calibrated through the time synchronization protocol to ensure that the timestamps of different data sources are aligned under the same time base, and the data with missing timestamps is supplemented by the interpolation algorithm; The spatial coordinate systems of different data sources are converted into the target coordinate system through the Geographic Information System (GIS), and the spatial data are projected and transformed using the coordinate transformation algorithm to ensure that the spatial data are aligned in the same coordinate system. For audio and video streaming data, key frames of the video data are detected and extracted through image processing algorithms, and compressed sensing technology is used to compress the key frames.

4. The method according to claim 1, wherein The multi-dimensional feature extraction of the unified data is specifically as follows: A feature dimensionality reduction method based on the combination of principal component analysis (PCA) and linear discriminant analysis (LDA) is used: Use PCA to perform linear transformation on high-dimensional data, determine the number of principal components based on the cumulative contribution rate threshold, and remove noise and redundant information; The data after PCA dimensionality reduction is then input into the LDA model to optimize the feature projection direction by maximizing the ratio of the inter-class scatter matrix and minimizing the intra-class scatter matrix.

5. The method according to claim 1, wherein The weight distribution of the dynamic weight distribution algorithm is dynamically adjusted based on the real-time index of the data source, the device credibility score and the historical error feedback, specifically: Among them, T i is the data real-time coefficient, T max is the maximum delay threshold allowed by the system, C i is the credibility coefficient based on the historical behavior evaluation of the device, E i is the historical fusion error value, E base is the benchmark error value, α, β, γ are configurable adjustment parameters.

6. The method according to claim 1, characterized in that The data quality assessment of the fused data packet specifically includes: The integrity of the data is verified by detecting missing data fields. If a field is detected to be missing, a retransmission request of the corresponding interface is triggered; Verify data consistency by comparing the attribute description difference threshold of the same entity in multi-source data; A correction instruction is generated based on the offset between the data generation timestamp and the system clock to verify the real-time nature of the data.

7. The method according to claim 1, characterized in that The specific method for dynamically adjusting the data acquisition interface transmission path according to the verification results is as follows: Build a reinforcement learning model based on deep Q network DQN; Taking historical verification results, transmission path usage, and current network status as state inputs, we design a reward function that gives positive rewards to stable and efficient transmission paths and negative rewards to paths that cause data quality issues. The network parameters are updated through the Q learning algorithm, and the experience replay mechanism is used to break the data correlation to achieve intelligent selection of the optimal data acquisition interface transmission path.

8. A multivariate data processing system, characterized in that: include: Data acquisition module: used to collect raw data from multiple data sources by calling multiple API interfaces to obtain multi-source heterogeneous data; Data preprocessing module: used to format the multi-source heterogeneous data according to preset format conversion rules, including timestamp alignment, spatial coordinate system conversion and key frame extraction; Data fusion module: used to extract multi-dimensional features from the unified data, generate spatiotemporally aligned feature vectors, perform cross-modal data fusion on the feature vectors based on a dynamic weight allocation algorithm, and generate a fused data packet; Data optimization module: used to evaluate the data quality of the fused data packet, including data integrity, consistency and real-time verification, and dynamically adjust the transmission path of the data acquisition interface based on the verification results.

9. An electronic device, characterized in that: include: processor; as well as, A memory arranged to store computer executable instructions, which, when executed, cause the processor to implement the steps of the multivariate data processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: Used to store computer-executable instructions, which, when executed, implement the steps of the multivariate data processing method according to any one of claims 1 to 7.

Images