Normalized data acquisition system based on big data

Abstract

The invention relates to the technical field of big data acquisition and processing, in particular to a normalized data acquisition system based on big data, which comprises a probe sensing module, a portrait generation module, an acquisition scheduling module, a data conversion module and a distribution routing module. The method comprises the following steps: sensing multi-source heterogeneous data in real time through a deployed distributed probe to generate an original data stream; the portrait generation module analyzes the data stream, extracts multi-dimensional features, and constructs a dynamic portrait of the data source; the acquisition scheduling module dynamically establishes and adjusts an acquisition logic framework according to the portrait so as to guide the collaborative acquisition behavior of the probe; the data conversion module performs pattern recognition and structure remodeling on the native data through a unified pipeline under the guidance of the framework, and outputs a data set with a unified pattern specification; and the distribution routing module injects the data into a data distribution network and delivers the data to a specified warehouse. According to the system, intelligent sensing and self-adaptive scheduling of a dynamic data source are realized, and real-time and efficient normalization processing of data from the source is completed.

Description

Technical Field

[0001] This invention relates to the field of big data acquisition and processing technology, and in particular to a normalized data acquisition system based on big data. Background Technology

[0002] In today's big data environment, data originates from numerous heterogeneous systems, devices, and applications, exhibiting significant differences in protocols, structures, and real-time states. Traditional data acquisition schemes typically rely on pre-configured static data source connection parameters and acquisition rules. The acquisition logic is fixed or periodically triggered, lacking real-time awareness of the inherent characteristics and state changes of the data source during runtime. This static acquisition model exhibits rigidity and lag when faced with sudden surges in data source traffic, changes in protocol behavior, or evolution of data structures.

[0003] The shortcomings of this technical approach directly lead to a disconnect between the data acquisition process and the actual state of the data source. The inability to accurately identify and adapt to the real-time characteristics of different data sources means that the acquisition process may miss key data or generate a large amount of invalid data, affecting the completeness and timeliness of data acquisition. Furthermore, the dispersed and differently configured acquisition points are difficult to coordinate globally, and resource allocation cannot be dynamically optimized according to real-time load, easily resulting in uneven load distribution or resource waste among acquisition nodes. At the same time, due to a lack of in-depth understanding of the characteristics of the data source, subsequent integration and normalization of heterogeneous data often faces challenges such as format confusion and semantic ambiguity, increasing the complexity and latency of the data processing chain.

[0004] There is a need for a technology that can sense the dynamic characteristics of data sources in real time and intelligently schedule collection behavior accordingly, in order to overcome the limitations of static configuration, achieve more efficient, accurate and adaptive data collection and front-end normalization, and provide high-quality and consistent data input for downstream data storage and analysis. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a normalized data acquisition system based on big data.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a normalized data acquisition system based on big data, comprising: The probe sensing module deploys multiple data probes in a distributed network environment. These multiple data probes monitor and sense data sources with different protocols and structures in real time, thereby forming a raw sensing data stream. The portrait generation module parses and captures features of the original perceptual data stream, extracts multi-dimensional feature information that represents the inherent characteristics of the data source, and generates a dynamic portrait of the data source based on the multi-dimensional feature information. The data acquisition scheduling module establishes a dynamically adjusted data acquisition logic framework based on the dynamic profile of the data source. This data acquisition logic framework is used to guide the collaborative acquisition behavior of the data probes. Under the guidance of the data acquisition logic framework, the data conversion module performs synchronous or asynchronous acquisition actions by multiple data probes, sending the acquired raw data into a unified data conversion pipeline. The unified data conversion pipeline performs pattern recognition and structural reshaping on the raw data, converting the heterogeneous raw data into a data set with a unified pattern specification. The distribution and routing module injects the data set with a unified pattern specification into a data distribution network with buffering and routing functions, and the data distribution network then delivers the data to the designated data warehouse.

[0007] As a further aspect of the present invention, the step of parsing and capturing features of the original perceptual data stream, extracting multi-dimensional feature information representing the intrinsic characteristics of the data source, and generating a dynamic profile of the data source based on the multi-dimensional feature information specifically includes: The header and payload information of the data packets are separated from the original sensing data stream. The header information is decoded according to the protocol, and the payload information is analyzed by shallow syntax analysis to obtain the basic descriptor of the data stream. The basic descriptor is processed by time serialization to construct a time series signal describing the evolution of data arrival interval, data packet size change, and data content entropy over time; Signal processing methods are used to filter the time series signal to suppress noise, and periodic patterns, trend components, and abrupt change information in the time series signal are detected. The detected periodic patterns, trend components, and mutation point information are fused to construct a composite feature vector that can reflect the activity, stability, and suddenness of the data source. Simultaneously, the data generation context implicit in the original sensing data stream is analyzed, which includes the physical deployment location of the data source, logical network topology, and interaction traces with the application layer. The composite feature vector is associated and mapped with the data generation context to form a data source description model that includes static attributes and dynamic behavioral features. The data source description model is the dynamic profile of the data source.

[0008] As a further aspect of the present invention, the step of using signal processing methods to filter the time series signal to suppress noise and detecting periodic patterns, trend components, and abrupt change information in the time series signal specifically includes: An adaptive filter is used to smooth the time series signal, and the parameters of the adaptive filter are automatically adjusted according to the local statistical characteristics of the time series signal. Spectral analysis is performed on smoothed time series signals to identify hidden periodic patterns by recognizing significant peaks in the spectrum. Meanwhile, the trend decomposition algorithm is used to separate the time series signal into long-term trend component, seasonal component and residual component; A mutation detection algorithm is applied to the residual components to identify short-term, drastic change points in the signal, which correspond to sudden events in the data source. The identified periodic pattern parameters, the mathematical expressions of the long-term trend components and seasonal components obtained by the trend decomposition algorithm, and the location and intensity information of the abrupt change points are recorded in a structured manner.

[0009] As a further aspect of the present invention, the step of establishing a dynamically adjustable data acquisition logic framework based on the dynamic profile of the data source, wherein the data acquisition logic framework is used to guide the collaborative acquisition behavior of the data probe, specifically includes: The data source description model contained in the dynamic profile of the data source is analyzed, and quantitative indicators about the expected lifespan of the data source, the estimated data value density, and the urgency of data collection are extracted from it. Based on the aforementioned quantitative indicators, a multi-objective optimization function is designed. This multi-objective optimization function aims to balance the integrity and timeliness of data collection with the load pressure on the network and the data source itself. Solving the multi-objective optimization function yields a set of optimal acquisition strategy parameters for dynamic profiling of different data sources. These optimal acquisition strategy parameters include acquisition frequency, acquisition depth, and acquisition parallelism. The optimal acquisition strategy parameters are matched with the real-time status information of the data probes, which includes processing capacity margin, network bandwidth usage, and current task queue length. Based on the matching results, a specific acquisition instruction set is assigned to each data probe, and the communication protocol and data format for task collaboration and status synchronization between data probes are specified. All data probes assigned to a set of acquisition instructions form a logically unified but physically distributed acquisition cluster under the constraints of the communication protocol and data format. The overall behavior rules of the acquisition cluster constitute the dynamically adjusted data acquisition logic framework.

[0010] As a further aspect of the present invention, the optimal acquisition strategy parameters are matched with the real-time status information of the data probes. The real-time status information of the data probes includes processing capacity margin, network bandwidth usage, and current task queue length. Based on the matching result, a specific acquisition instruction set is assigned to each data probe, specifically including: Establish a resource matching matrix, where the rows of the resource matching matrix represent data probes and the columns represent the dimensions of the collection task requirements defined by the optimal collection strategy parameters; The real-time status information of each data probe is quantified into a resource supply vector, and the optimal acquisition strategy parameters of each acquisition task are quantified into a resource demand vector. Calculate the fit score between each resource supply vector and all resource demand vectors. The fit score takes into account the capacity matching degree, load balancing degree and geographical proximity. A task allocation algorithm is used to assign collection tasks to the most suitable data probes based on the suitability score, thus forming a preliminary task allocation scheme. The degree to which the preliminary task assignment scheme achieves the overall goal of the data acquisition logic framework is evaluated. If the degree of achievement is lower than the threshold, the parameters of the task assignment algorithm are adjusted or constraints are added for iterative optimization until the degree of achievement meets the requirements. The finalized task assignment scheme is compiled into a set of acquisition instructions that can be executed by each data probe. The acquisition instruction set includes the target data source identifier, acquisition protocol, parameter configuration, and task scheduling plan.

[0011] As a further aspect of the present invention, the task allocation algorithm is adopted to assign the acquisition task to the most suitable data probe based on the suitability score, forming a preliminary task allocation scheme, specifically including: The fit scores between the resource supply vector of each data probe and the resource demand vector of each acquisition task are arranged into a fit score matrix. In the fit score matrix, the data probe with the highest fit score for each acquisition task is selected as the candidate execution probe for the acquisition task. Check the number of acquisition tasks assigned to each candidate execution probe. If the number of acquisition tasks assigned to a candidate execution probe exceeds the upper limit that its processing capacity margin can bear, then mark the acquisition task with the lowest fit score on the candidate execution probe as a task to be reassigned. For each task to be reassigned, find the next data probe with the highest fit score. If the data probe is not currently assigned a task or the number of assigned tasks has not reached the upper limit, then assign the task to be reassigned to the data probe. Repeat the checking and redistribution process until all acquisition tasks are assigned to a single data probe, and the number of acquisition tasks assigned to each data probe does not exceed its processing capacity margin limit. At this point, a preliminary task allocation scheme is obtained.

[0012] As a further aspect of the present invention, under the guidance of the data acquisition logic framework, multiple data probes perform synchronous or asynchronous acquisition actions, sending the acquired raw data into a unified data transformation pipeline, specifically including: Each data probe establishes a connection session with the target data source according to its assigned set of acquisition instructions, and performs data extraction operations according to the specified acquisition frequency, acquisition depth and acquisition parallelism. During the data extraction process, the data probe continuously monitors the session status and data stream quality, and feeds back the monitored session status and data stream quality information to the dynamically adjusted data acquisition logic framework in real time. The dynamically adjusted data acquisition logic framework dynamically adjusts the acquisition instruction set of relevant data probes based on the feedback session status and data stream quality information, forming a closed-loop control. The data probe caches the successfully extracted raw data blocks locally and attaches metadata tags, which record the source, collection time, session identifier, and data format fingerprint of the data blocks. When the locally cached data block reaches a certain threshold or a transmission instruction is received, the data probe encapsulates the data block with metadata tags into a transmission unit according to the communication protocol and data format. The transmission units generated by multiple data probes are pushed in an orderly manner to the entry buffer of the unified data conversion pipeline through an asynchronous message mechanism or a synchronous call interface.

[0013] As a further aspect of the present invention, the unified data transformation pipeline performs pattern recognition and structural reshaping on the raw data, transforming the heterogeneous raw data into a data set with a unified pattern specification, specifically including: The unified data conversion pipeline's inlet buffer receives transmission units from multiple data probes, decapsulates the transmission units, and separates the original data blocks and their accompanying metadata tags. Based on the data format fingerprint in the metadata tag, a pre-built or dynamically loaded format parser is invoked to perform deep parsing of the original data block, identifying the internal structure, fields, types and constraints of the data; The identified data's internal structure, fields, types, and constraints are matched and mapped to a globally unified schema definition library, which defines the target specification's data model. Based on the matching and mapping results, the data in the original data block is cleaned, transcoded, split, merged, or the derived fields are calculated to eliminate ambiguity, fill missing information, correct errors, and meet the constraints of the target data model. After the data has been cleaned, transcoded, split, merged or derived fields have been calculated, it is reorganized into new data records according to the target data model, and a globally unique identifier and version stamp are generated for each new data record. All new data records that conform to the target data model are batched into data blocks, which together constitute a data set with a unified schema specification.

[0014] As a further aspect of the present invention, the operation of cleaning, transcoding, splitting, merging, or calculating derived fields of the data in the original data block based on the matching and mapping results specifically includes: For data with missing fields or abnormal value ranges, data is filled in according to the default value rules defined in the globally unified pattern definition library or the filling model learned from historical data. For text data with inconsistent encoding formats, a conversion table or character set detection technology is used to uniformly convert them into the specified standard encoding. For complex data that is too deeply nested or has an overly complex structure, it should be split into multiple related simple data records in accordance with the flattening requirements of the target data model. For data that logically belong to the same entity but are physically scattered in different original data blocks, they are merged according to entity parsing rules to generate a complete entity record; Based on business rules or statistical analysis requirements, new derived fields are generated by using the basic fields in the original data block through predefined functions or scripts. All operations involving cleaning, transcoding, splitting, merging, or calculating derived fields are recorded in a data tracing log, which is associated with the final generated new data record.

[0015] As a further aspect of the present invention, injecting the data set with a unified schema specification into a data distribution network with buffering and routing functions, and having the data distribution network ultimately deliver the data to a designated data warehouse, specifically includes: The data set with a unified pattern specification is divided into appropriately sized data fragments, and each data fragment is attached with a routing label containing the target data warehouse address, data priority, and delivery time requirements; Based on the data priority and delivery time requirements in the routing labels, data fragments are allocated to transmission queues of different service levels in the data distribution network; Routing nodes in the data distribution network continuously probe the network path status leading to the target data warehouse, and the network path status includes latency, packet loss rate, and available bandwidth. Combining the service level of the transmission queue with the real-time detected network path status, the routing node dynamically selects the optimal next-hop node or transmission link for each data fragment. During transmission, if network congestion or path failure is detected, the routing node can re-route the data fragments to an alternative path according to a preset strategy, or temporarily store them in local persistent storage to await recovery. When a data fragment successfully arrives at the receiving gateway of the target data warehouse, the receiving gateway verifies the integrity of the data fragment and sends an acknowledgment. After being unpacked and reassembled, the data fragment is stored in the designated storage location of the data warehouse.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: By performing real-time analysis and feature capture on the raw, perceived data stream, the system extracts and continuously updates multi-dimensional feature information reflecting the intrinsic state of the data source, forming a dynamic profile. This enables the system to transcend the descriptive level of static metadata and grasp the behavioral patterns, structural characteristics, and load status of the data source in real time. Based on this deep understanding of the dynamic profile, the system's perception of the data source transforms from a "known fixed endpoint" to a "perceptible dynamic entity," thus providing a real-time and reliable basis for the accurate formulation of collection strategies and avoiding blind spots or biases caused by information lag or omissions.

[0017] Based on dynamic profiles, a data acquisition logic framework is established and adjusted in real time to guide the collaborative work of distributed probes. The initiation of acquisition tasks is driven by dynamic data source status profiles, intelligently determining the timing, frequency, priority, and synchronous collaboration relationships between multiple probes based on the importance, activity level, and data flow characteristics of the data source. This adaptive scheduling mechanism allows for flexible allocation of acquisition resources, concentrating resources to ensure acquisition quality when high-value data is available, and reducing acquisition frequency to conserve resources when the data source is inactive. The acquisition process transforms from a predefined, fixed execution plan into a dynamic, collaborative, and efficient intelligent behavior that responds to changes in the data source environment.

[0018] The collected raw data is fed into a unified data transformation pipeline. Based on the deep understanding of the data sources accumulated during the earlier perception and scheduling phases, targeted pattern recognition and structural reshaping are performed. Because the system has a dynamic understanding of the data source characteristics before and during collection, the transformation process can more accurately parse the semantics and structure of heterogeneous data, efficiently mapping them into a data set with a unified pattern standard. This coherent technical path from "perception" to "collection" to "transformation" ensures that data is oriented towards standardized processing from the source, improving the efficiency of data integration and the consistency and quality of the output data. Attached Figure Description

[0019] Figure 1 This is a timing diagram of the normalized data acquisition system based on big data described in this invention. Figure 2 This is a flowchart of time series signal processing and feature extraction. Figure 3 To compare the load changes of the data probe before and after task allocation; Figure 4 A graph showing the dynamic change of sampling frequency over time for different probes; Figure 5 A heatmap showing the fit between the data probe and the acquisition task. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] See Figure 1This invention relates to a normalized data acquisition system based on big data, and will elaborate on the specific implementation method by combining the functions and collaborative relationships of the various modules of the system. The system operates in a distributed network environment, and its core purpose is to achieve unified, efficient, and adaptive acquisition and normalization processing of multi-source heterogeneous data. The overall implementation scheme of the system is as follows: Through a widely deployed probe sensing module, data sources with different protocols and structures are monitored and sensed in real time to form a raw sensing data stream. The profile generation module parses and captures features of the data stream, extracts multi-dimensional feature information, and generates a dynamic profile of the data source. Based on this dynamic profile, the acquisition scheduling module establishes a dynamically adjusted data acquisition logic framework to guide the collaborative acquisition behavior of the data probes. Under the guidance of the data acquisition logic framework, multiple data probes perform synchronous or asynchronous acquisition actions and send the acquired raw data to a unified data conversion pipeline in the data conversion module. This unified data conversion pipeline performs pattern recognition and structural reshaping on the raw data, converting heterogeneous data into a data set with a unified pattern specification. The distribution and routing module injects this data set into a data distribution network with buffering and routing functions, and the data distribution network finally delivers the data to the designated data warehouse.

[0023] In one embodiment of the present invention, see [reference] Figure 2 In practical implementation, the profile generation module is deployed on distributed data processing nodes. This module receives the raw sensing data stream from the probe sensing module. The raw sensing data stream contains a sequence of data packets continuously sent from various data sources in the network environment, with different protocol encapsulations and payload structures. The profile generation module parses and captures features from the raw sensing data stream as follows: The module separates the header and payload information of each data packet from the raw sensing data stream in real time; it performs protocol decoding on the header information to determine the communication protocol type, source address, destination address, and port information; it performs shallow syntax analysis on the payload information to identify the approximate structure of the data, such as whether it is a key-value pair, a JSON object, or an XML document fragment; through the above parsing, it obtains the basic descriptor of the data stream, including the protocol type, data packet size, timestamp, and coarse data structure identifier.

[0024] In practical implementation, the profile generation module organizes the obtained basic descriptors in chronological order to construct time-series signals describing the evolution of data arrival intervals, packet size variations, and data content entropy over time. For example, for a data source from an industrial sensor network, the data arrival interval time-series signal reflects the periodicity or event-triggered nature of the sensor-reported data; the packet size time-series signal characterizes the stability of the transmitted data volume; and the content entropy time-series signal, generated by real-time entropy calculation of the payload content, can indirectly reflect the randomness or information density of the data content. These time-series signals collectively constitute a quantitative representation of the data source's behavioral patterns.

[0025] In some embodiments, the specific operation of filtering a time series signal using signal processing methods to suppress noise includes employing an adaptive filter. The adaptive filter dynamically adjusts its filter coefficients based on the local statistical characteristics of the input time series signal to optimally filter out background noise. One implementation uses a least mean square algorithm to update the filter weights, and the weight update process can be expressed as follows:

[0026] in: Represents the moment The filter weight vector, It is the step size parameter that controls the convergence speed. It is a moment The error signal between the expected response and the actual output of the filter. It is a moment The input signal vector. Through this process, the adaptive filter can effectively smooth out random jitter in time series signals of data arrival intervals or packet sizes without masking the true trend changes.

[0027] In practice, the detection of periodic patterns, trend components, and abrupt changes in time-series signals is carried out in parallel. Fast Fourier Transform (FFT) or power spectral density estimation is performed on the time-series signal after adaptive filter smoothing. By identifying significant energy peaks and their corresponding frequencies in the spectrogram, hidden periodic patterns in data generation or transmission behavior are discovered. Simultaneously, algorithms such as seasonal trend decomposition are used to separate the time-series signal into long-term trend components, seasonal components, and residual components. The long-term trend component reveals the long-term trend of data source activity, while the seasonal component corresponds to fixed periodic fluctuations. Abrupt change detection algorithms based on sliding window statistics or cumulative sums are applied to the separated residual components to identify short-term, drastic changes in the signal. These changes correspond to sudden events or state transitions in the data source.

[0028] It is understandable that fusing detected periodic patterns, trend components, and abrupt change information is the foundation for constructing a composite feature vector. The profile generation module structurally records and encodes the frequency and amplitude parameters of periodic patterns, the slope of trend components, and the location and intensity of abrupt changes into numerical values. For example, the detected main period length, the sign and magnitude of the trend slope, and the number and average intensity of abrupt changes per unit time are combined into a multi-dimensional composite feature vector in a predetermined order. This composite feature vector mathematically characterizes the dynamic behavior of the data source in terms of activity, stability, and suddenness. In some embodiments, analyzing the data generation context implicit in the raw perceived data stream is a parallel processing branch. The acquisition of the data generation context does not rely on deep parsing of packet payloads, but rather on analyzing network flow characteristics, routing table information, and interacting with logs or configuration interfaces of upper-layer application systems. For example, by tracing the mapping relationship between IP addresses and network topology, the physical deployment location of the data source, such as a data center, edge gateway, or mobile terminal, can be determined; by analyzing the communication peers and hop count information of the data stream, its logical network topology can be inferred; and by parsing the application layer protocol identifier or associated application performance management data in the packet header, traces of the interaction between the data source and the application layer can be captured.

[0029] Optionally, the composite feature vector can be mapped to the data generation context to form a data source description model. The profile generation module maintains a context feature mapping table, associating static attributes such as physical deployment location and logical role with dynamic behavioral indicators in the composite feature vector. For example, for a data source identified as an "edge temperature sensor," the high periodicity and low burstiness characteristics exhibited in its composite feature vector will be bound to context labels such as "edge device" and "monitoring data," jointly forming a data source description model containing static attributes and dynamic behavioral features. This data source description model is output as the dynamic profile of the data source to the acquisition and scheduling module. It can be understood that the generation of the dynamic profile is a continuous iterative process. As new raw sensing data streams are continuously input, the time series signal is updated, the signal processing method will re-execute filtering, decomposition, and detection, the composite feature vector will be recalculated, and the data generation context may also be updated due to changes in network status, thus causing the data source description model, i.e., the dynamic profile, to be dynamically refreshed accordingly to reflect the latest characteristics of the data source in real time.

[0030] In one embodiment of the present invention, in a specific implementation, the acquisition scheduling module receives a dynamic profile of the data source from the profile generation module. The dynamic profile is encapsulated and transmitted in the form of a data source description model. The implementation method of the acquisition scheduling module to establish a dynamically adjusted data acquisition logic framework is as follows: The acquisition scheduling module parses the data source description model and extracts quantitative indicators regarding the expected lifespan of the data source, the estimated data value density, and the urgency of data acquisition. The expected lifespan indicator is inferred based on the historical stability of the data source, maintenance logs, and device type. The estimated data value density indicator is calculated based on the business type of the data source, the information entropy of the composite feature vector in the dynamic profile, and the historical data utilization rate. The urgency of data acquisition is assigned a value by comprehensively considering the time sensitivity of the data and the frequency and intensity of the information of sudden changes in the dynamic profile.

[0031] In practical implementation, a multi-objective optimization function is designed based on the extracted quantitative indicators. This function aims to balance the completeness and timeliness of data collection with the load on the network and the data source itself. The multi-objective optimization function integrates multiple conflicting objectives into a single mathematical expression, which can be represented as:

[0032] in: Represents the overall optimization target value. It is a sub-objective function characterizing the completeness of data collection; a higher value indicates fewer data omissions. It is a sub-objective function characterizing the timeliness of data collection; the higher its value, the lower the delay from data generation to collection. It is a sub-objective function that characterizes the load pressure on the network and data source; a higher value indicates lower resource consumption. , and These are non-negative weight coefficients corresponding to the three sub-targets. The specific values ​​of the weight coefficients are dynamically set by the acquisition and scheduling module based on the global strategy and the relative importance of the quantitative indicators in the dynamic profile of the current data source.

[0033] In some embodiments, solving the multi-objective optimization function yields a set of optimal acquisition strategy parameters for dynamic profiling of different data sources. The solution process employs an optimization algorithm. The acquisition scheduling module instantiates an optimization problem for each data source dynamic profiling, with decision variables including acquisition frequency, acquisition depth, and acquisition parallelism. Acquisition frequency determines the number of acquisitions initiated per unit time, acquisition depth defines the level of detail or historical backtracking range of data acquired in each acquisition operation, and acquisition parallelism indicates the upper limit of the number of probes that can simultaneously initiate acquisitions for that data source. By running optimization algorithms such as gradient descent or genetic algorithms, and under the premise of satisfying preset constraints, the optimal multi-objective optimization function is found. The optimal combination of sampling frequency, sampling depth, and sampling parallelism is the optimal sampling strategy parameters for that data source. Matching these optimal sampling strategy parameters with the real-time status information of the data probes is a prerequisite for task assignment. The sampling scheduling module maintains an in-memory database containing the status of all online data probes. The real-time status information of the data probes is continuously reported via a heartbeat mechanism. Processing capacity margin is represented by the current CPU and memory idle percentages, network bandwidth utilization is represented by the current traffic rate entering and leaving the network interface, and the current task queue length is represented by the number of sampling commands waiting to be executed. The matching process requires translating the optimal sampling strategy parameters into resource requirements for the probes and comparing them with the real-time resource supply of the probes.

[0034] Optionally, a specific collection instruction set is assigned to each data probe based on the matching results. The collection instruction set is an executable command sequence or configuration file. The collection instruction set explicitly includes the network identifier or access address of the target data source, the specific collection protocol to be used, the parameter configuration set according to the optimal collection strategy parameters, and the task scheduling plan. The task scheduling plan specifies the start time, execution cycle, and timeout handling rules for the collection task. The collection instruction set is sent to the corresponding data probe through a secure communication channel. Upon receiving the collection instruction set, the data probe parses it and loads and executes it. In some embodiments, a communication protocol and data format are specified for task coordination and state synchronization between data probes. The collection scheduling module defines a message-based communication protocol. Message types include task status broadcasts, data block transmission coordination requests, and load balancing negotiations. The data format is defined using serialization frameworks such as Protocol Buffers or Apache Avro to ensure that data probes implemented in different programming languages ​​can correctly exchange information. This communication protocol and data format specification serves as the standard for interaction within the collection cluster.

[0035] In practice, all data probes assigned acquisition command sets form an acquisition cluster under the constraints of communication protocols and data formats. Logically, the acquisition cluster presents itself as a unified task execution entity, subject to overall coordination by the acquisition scheduling module. Physically, the data probes are distributed and deployed across different network nodes. The acquisition cluster operates according to the overall rules of a dynamically adjusted data acquisition logic framework. These rules are reflected in the continuous optimization function solving, parameter matching, and command assignment loops performed by the acquisition scheduling module, enabling the entire acquisition process to adapt to changes in data source characteristics and network environment.

[0036] In one embodiment of the present invention, in a specific implementation, the process of matching the optimal acquisition strategy parameters with the real-time status information of the data probes is executed by a dedicated matching engine in the acquisition scheduling module. The matching engine establishes a resource matching matrix. The rows of the resource matching matrix represent all registered and available data probes in the system, and the columns of the resource matching matrix represent the acquisition task requirement dimensions defined by the optimal acquisition strategy parameters. The acquisition task requirement dimensions include at least the minimum requirements for the computing power of the data probes, network throughput requirements, required acquisition protocol support, and geographical region preferences. The matching engine quantizes the real-time status information of each data probe into a multi-dimensional resource supply vector. The elements of the resource supply vector include the processing capacity margin in units of millions of instructions per second, the inverse value of available network bandwidth occupancy in units of megabits per second, and the current task queue length slack in units of the number of probes. At the same time, the matching engine quantizes each acquisition task to be allocated into a resource requirement vector according to its optimal acquisition strategy parameters. The elements of the resource requirement vector are mapped to the column dimensions of the resource matching matrix, including the estimated computing intensity required by the acquisition task, the expected data traffic, the required protocol type, and the preferred data probe deployment area.

[0037] In practice, a fit score is calculated between each resource supply vector and all resource demand vectors. This fit score is a scalar value used to quantify the suitability of a specific data probe for performing a specific data collection task. The calculation of the fit score comprehensively considers capability matching, load balancing, and geographical proximity. Capability matching is obtained by calculating the cosine similarity or the inverse of the Euclidean distance between the resource supply vector and the resource demand vector across each capability dimension. Load balancing is obtained by evaluating the variance or Gini coefficient change of the overall load of all data probes after assigning the data collection task to that data probe. Geographical proximity is obtained by querying a network topology database to calculate the network latency or routing hops between the data probe and the target data source. The comprehensive formula for calculating the fit score can be expressed as:

[0038] in: Representative data probe For data collection tasks The fit score, This represents the normalized ability matching degree; a higher value indicates a better ability match. This represents the normalized load balancing degradation level; a lower value indicates a smaller impact on the overall load balancing. Network distance represents a normalized measure of geographic proximity; a lower value indicates a closer network distance. , and It is a weighting coefficient used to adjust the relative importance of the three factors, and satisfies After the calculation is completed, all This forms the fitness score matrix.

[0039] In some embodiments, a task allocation algorithm is used to assign collection tasks to the most suitable data probes based on fit scores to form a preliminary task allocation scheme. The task allocation algorithm follows an iterative greedy and adjustment strategy. The matching engine assigns each collection task to a matching task in the fit score matrix. Select fit score Highest data probe This data probe Mark as a data collection task The matching engine checks the total number of acquisition tasks assigned to each candidate execution probe and compares this number with the upper limit of parallel tasks that the processing capacity margin can support in the real-time status information of the data probe. If the number of acquisition tasks assigned to a candidate execution probe exceeds its processing capacity margin upper limit, the matching engine marks the acquisition task with the lowest fit score among all the acquisition tasks assigned to that candidate execution probe as a task to be reassigned.

[0040] It is understandable that finding the next suitable execution probe for each task to be reassigned is the core adjustment step of the task allocation algorithm. For each task to be reassigned, the matching engine excludes the original candidate execution probes in its corresponding fit score matrix column and finds the data probe with the second highest fit score. It checks whether the number of tasks currently assigned to this new data probe has reached its processing capacity margin limit. If the data probe is not currently assigned any tasks or the number of assigned tasks has not reached the limit, the matching engine assigns the task to be reassigned to this new data probe and updates the task allocation record. If this new data probe is also full, it continues to search for the next data probe with a higher fit score until a data probe that can receive the task is found, or all possible data probes are traversed.

[0041] Optionally, the checking and reallocation process is repeated until the termination condition is met. After each round of adjustment, the matching engine recalculates the number of assigned tasks for all data probes and checks again for overloaded data probes. If data probes are still overloaded, a new round of reallocation is initiated. This process is repeated until all acquisition tasks are assigned to a single data probe, and the number of acquisition tasks assigned to each data probe does not exceed its processing capacity margin limit. The complete mapping set obtained at this point constitutes the preliminary task allocation scheme. In some embodiments, the achievement of the preliminary task allocation scheme to the overall goal of the data acquisition logic framework is evaluated based on the multi-objective optimization function initially designed by the acquisition scheduling module. The matching engine substitutes the preliminary task allocation scheme into the multi-objective optimization function to calculate the comprehensive performance of the three sub-objectives of data acquisition integrity, timeliness, and load pressure under the current allocation scheme, obtaining a quantified achievement value. The matching engine compares this achievement value with a preset performance threshold. If the achievement is lower than the threshold, optimization adjustment is triggered. The optimization adjustment includes adjusting the weighting coefficient of the fit score calculation formula in the task allocation algorithm. , , Alternatively, new constraints can be added during the task allocation process, such as forcibly guaranteeing the upper limit of the acquisition delay for certain high-priority tasks. Then, the entire task allocation algorithm can be rerun based on the new parameters or constraints to form a new preliminary task allocation scheme and evaluate it again. This iterative optimization process continues until the achievement of the generated task allocation scheme meets the preset threshold requirements.

[0042] In practice, the final task allocation scheme is compiled into a set of collection instructions executable by each data probe. The matching engine generates a unique instruction list for each data probe based on the final task allocation scheme. This list contains detailed descriptions of all collection tasks assigned to that data probe. Each collection task description is compiled into a collection instruction set, which is a structured data object or file that explicitly includes the target data source identifier, collection protocol, parameter configuration, and task scheduling plan. The target data source identifier can be an IP address, URL, or a globally unique resource identifier. The collection protocol specifies the specific type and version, such as HTTP, FTP, Kafka, or a dedicated industrial protocol. The parameter configuration is refined according to the optimal collection strategy parameters into specific operable settings such as connection timeout, request header information, data filtering conditions, and page size. The task scheduling plan defines the task's start time, fixed-interval cycle, allowed execution time window, and failure retry strategy. The compiled collection instruction set is then distributed to the corresponding data probes through the command channel established between the collection scheduling module and the data probes.

[0043] See Figure 3 This chart compares the load changes of the data probes before and after task allocation, combining bar and line charts to clearly show the load and capacity limits of the 10 data probes. The load of all probes after task allocation (purple bars) is higher than before allocation (red bars), indicating that the task allocation did indeed add new data collection tasks to the probes. After task allocation, the load of each probe is concentrated in the range of 10-15 tasks, with a relatively even distribution and no extreme overload. The red dashed line represents the maximum load limit of each probe, determined by its processing capacity margin. The final load of all probes did not exceed this limit, verifying the safety and rationality of the task allocation algorithm. Probe 6 showed the most significant load increase, from approximately 13 tasks to 15 tasks, approaching its maximum load limit (red dashed line), making it the node under the most pressure in this allocation.

[0044] In one embodiment of the present invention, in a specific implementation, after receiving the collection instruction set issued by the collection scheduling module, the data probe begins to execute synchronous or asynchronous collection actions. Each data probe establishes a network connection session with the target data source according to the target data source identifier, collection protocol and parameter configuration contained in the collection instruction set. The connection session may be established based on TCP, UDP or application layer protocol handshake. The data probe initiates data extraction operations periodically or under specific triggering conditions according to the collection frequency specified in the collection instruction set. The data range extracted each time is determined according to the collection depth parameter, whether it is the full amount, incremental or a snapshot of a specific time window. The number of concurrent connections or the number of parallel processing threads are controlled according to the collection parallelism parameter to extract data from different partitions or endpoints of the data source simultaneously.

[0045] In practice, the data probe continuously monitors session status and data stream quality during data extraction. Session status monitoring includes checking connection integrity, authentication validity, and request response time within acceptable limits. Data stream quality monitoring includes checking the integrity of received data blocks, sequence number continuity, and whether they conform to the expected data format header identifiers. The data probe encapsulates the monitored session status and data stream quality information into status report messages in real time. These messages are sent to the dynamically adjusted data acquisition logic framework maintained by the acquisition scheduling module via a dedicated feedback channel. The dynamically adjusted data acquisition logic framework includes a feedback analyzer that parses the status report messages. When session anomalies or data stream quality degradation are detected, the feedback analyzer dynamically adjusts the acquisition instruction set of the relevant data probes according to preset strategies. For example, it may reduce the acquisition frequency of problematic data sources, switch to backup acquisition protocols, or temporarily adjust the acquisition depth, forming a closed-loop control from execution to monitoring to adjustment.

[0046] In some embodiments, the data probe locally caches the successfully extracted raw data blocks and attaches metadata tags. The raw data blocks are unconverted binary or text data units obtained directly from the data source. The attached metadata tag is a data structure bound to the raw data block, recording the data block's source, acquisition time, session identifier, and data format fingerprint. The data format fingerprint is a string of features generated by a hash function, used to uniquely identify or closely match a specific data format or structure followed by the raw data block, such as a specific pattern of a JSON array or a specific version of a binary protocol. The local cache of the data probe uses persistent storage media to prevent data loss due to process restarts, and the caching strategy is managed according to first-in-first-out (FIFO) or data priority rules.

[0047] It is understandable that when the locally cached data blocks reach a certain threshold or a transmission instruction is received from the acquisition scheduling module, the data probe initiates the data block transmission process. According to the communication protocol and data format specified by the acquisition scheduling module, the data probe encapsulates the original data blocks with metadata tags into transmission units. The encapsulation process includes adding a sequence number, total length verification, and destination address information to the transmission units. Transmission units generated by multiple data probes are pushed in an orderly manner to the entry buffer of a unified data conversion pipeline through an asynchronous message mechanism or synchronous call interface. The asynchronous message mechanism is typically implemented using message queue middleware, such as Kafka or RabbitMQ. Each data probe acts as a producer, publishing messages to a specified topic, while the unified data conversion pipeline acts as a consumer, subscribing to that topic. The synchronous call interface may send the transmission units directly to the receiving endpoint of the entry buffer via HTTPPOST or gRPC calls.

[0048] In practice, the unified data conversion pipeline's ingress buffer receives transmission units from multiple data probes. The ingress buffer is a queue system with flow control and load protection functions. It decapsulates the received transmission units, separating the original data blocks and their accompanying metadata tags. Based on the data format fingerprint in the metadata tag, the ingress buffer calls a pre-set or dynamically loaded format parser. The mapping relationship between the data format fingerprint and the format parser is stored in a registry, see Table 1.

[0049] Table 1: Data Format Fingerprint and Format Parser Mapping Table

[0050] Pre-built parsers are loaded when the unified data conversion pipeline starts, while dynamically loaded parsers are downloaded and instantiated from the remote parser repository when an unknown data format fingerprint is encountered for the first time. The invoked format parser performs deep parsing on the original data block, identifying the internal structure, fields, types and constraints of the data, such as parsing JSON strings into attribute trees and splitting binary streams into field sequences according to protocol specifications.

[0051] In some embodiments, the identified data's internal structure, fields, types, and constraints are matched and mapped to a globally unified schema definition library. This library defines the target specification's data model, described in a schema language such as AvroSchema or a Protobuf.proto file. The matching and mapping process involves semantic alignment of field names, data type conversion compatibility checks, and value range constraint verification. This process can be formalized as an optimization problem of finding the optimal mapping relationship. Its objective function is to maximize the semantic similarity of field mappings and the safety of data type conversions, while minimizing information loss. This function can be expressed as:

[0052] in: Representing the overall mapping quality score, the summation symbol iterates through all candidate mapping pairs between the source and target fields. , It is the source field With target field The semantic similarity function between them These are the weight coefficients of the mapping pair. The summation operator iterates through all source fields that need to be discarded or processed because they cannot be mapped. , Is it discarding the source field? The resulting information loss measurement It is the corresponding penalty coefficient; the matching engine calculates and finds the penalty coefficient. Maximize the mapping scheme. Optionally, based on the matching and mapping results, perform a series of reshaping operations on the data in the original data block. Reshaping operations include cleaning, transcoding, splitting, merging, or calculating derived fields. Cleaning operations target ambiguous, missing, or erroneous data, such as converting non-standard Boolean representations "yes / no" to "true / false". Transcoding operations convert text data from one character encoding to another, such as from GBK to UTF-8. Splitting operations flatten nested complex structures into multiple independent fields or records. Merging operations aggregate logically related but physically scattered data fragments based on the primary key. Calculating derived fields generates new fields using existing fields according to business rules, such as calculating the "age" field from the "date of birth" field.

[0053] It is understandable that the data, after being processed by the reshaping operation, is reorganized into new data records according to the target data model. A unified data transformation pipeline assigns a globally unique identifier to each newly generated data record. The globally unique identifier is usually generated by combining the machine identifier, timestamp, and sequence number. At the same time, a version stamp is added to the data record to identify the version of the target data model that the data record follows. All new data records that conform to the target data model are batch-assembled into data blocks. The data block is the basic unit output by the unified data transformation pipeline. These data blocks together constitute a data set with a unified schema specification. The data set is structurally completely in line with the globally unified schema definition library, thus providing a consistent interface for downstream distribution and storage.

[0054] See Figure 4 This is a graph showing the dynamic changes in the sampling frequency of different probes over time, clearly illustrating the periodic scheduling characteristics and collaborative patterns of the probes. The sampling frequency of all probes exhibits a clear sinusoidal periodic fluctuation, with a period of approximately 8 minutes, reflecting the regularity of the system's dynamic adjustment of the sampling strategy. Probe 4 shows the smallest fluctuation amplitude, with a stable frequency of 16-20 times / minute, indicating that its data source is relatively stable and has a low demand for dynamic adjustment of the sampling frequency. The high-frequency fluctuation of probe 3 suggests that the activity of its corresponding data source varies greatly, requiring the system to adjust the sampling strategy more frequently to adapt to dynamic changes. Clearly displaying the periodic fluctuations and phase coordination relationship of the probe sampling frequency directly verifies whether the dynamic sampling logic framework achieves load balancing and resource optimization as expected.

[0055] In one embodiment of the present invention, in a specific implementation, the operation of cleaning, transcoding, splitting, merging, or calculating derived fields of the data in the original data block according to the matching and mapping results is performed by the data reshaping engine in a unified data transformation pipeline. For data with missing fields or abnormal value ranges, the data reshaping engine performs data completion according to the default value rules defined in the globally unified pattern definition library or the filling model learned from historical data. The default value rules clearly specify the static values ​​to be used when a specific field is missing or exceeds a reasonable range, such as filling the missing "gender" field with "unknown". The filling model learned from historical data dynamically generates filling values ​​by analyzing the correlation between fields, such as inferring the missing "region" information of the same user through other records.

[0056] In practical implementation, for text data with inconsistent encoding formats, the data reshaping engine converts it to a specified standard encoding by querying a pre-set transcoding table or running character set detection technology. The pre-set transcoding table stores the mapping relationship from multiple source encodings to the target UTF-8 encoding. The character set detection technology determines the source encoding format probabilistically by analyzing the statistical characteristics of the byte sequence and calls the corresponding transcoding routine. For complex data that is too deeply nested or has an overly complex structure, the data reshaping engine splits it into multiple related simple data records according to the flattening requirements of the target data model. For example, it splits a JSON document containing multi-level sub-objects into a main table and several subordinate tables related by foreign keys, and establishes association identifiers for the records generated by the splitting.

[0057] In some embodiments, for data that logically belong to the same entity but are physically scattered across different original data blocks, the data reshaping engine merges them according to entity resolution rules to generate a complete entity record. Entity resolution rules define how to identify and link fragment data pointing to the same entity through one or more key fields. The data reshaping engine maintains an entity resolution rule base, which contains the identification and merging logic for various entities such as customers, devices, and orders. When performing the merging, the data reshaping engine extracts data fragments belonging to the same entity from different original data blocks, resolves conflicts and merges fields according to the rules, and finally outputs a complete, non-duplicate entity record. According to business rules or statistical analysis requirements, the data reshaping engine uses the basic fields in the original data blocks to calculate and generate new derived fields through predefined functions or scripts. For example, it calculates the "equivalent base currency amount" based on the "transaction amount" and "transaction currency" combined with the real-time exchange rate, or calculates the "age" based on the "date of birth".

[0058] It is understandable that all operations involving cleaning, transcoding, splitting, merging, or calculating derived fields are systematically recorded in the data source log. The data source log is a structured audit pipeline. Each log record contains the operation type, operation timestamp, the identifier of the original data block to which it was applied, the identifier of the applied rule or model, snapshots of field values ​​before and after the operation, and the identifier of the data reshaping engine node that performed the operation. The data source log is associated with the final generated new data record through a unique tracking identifier, thereby ensuring that the final data can be traced back to its original form and all the transformation steps it has undergone.

[0059] In practical implementation, the distribution routing module injects a data set with a unified pattern specification into a data distribution network with buffering and routing functions. The distribution routing module divides the data set into appropriately sized data fragments. The size of the data fragments is optimized according to the receiving capacity of the target data warehouse and the network MTU. A routing label is attached to each data fragment. The routing label is a structured information block embedded in the fragment header or as an independent meta data packet. The routing label contains the target data warehouse address, data priority, and delivery timeliness requirements. The data priority can be three levels: "high", "medium", and "low". The delivery timeliness requirements define time constraints such as "real-time", "minute-level", and "hour-level".

[0060] In some embodiments, data fragments are allocated to transmission queues of different service levels in the data distribution network according to the data priority and delivery timeliness requirements in the routing labels. The data distribution network maintains multiple transmission queues with different service guarantee levels, such as priority queues that guarantee low latency, weighted fair queues that guarantee bandwidth, and best-effort ordinary queues. The routing decision-maker of the distribution routing module checks the routing label of each data fragment and injects it into the corresponding transmission queue according to a predefined policy. The decision function of the injection policy can consider various factors:

[0061] in: The score represents the priority score for queue selection. It is a numerical mapping representing data priority. It is a numerical time constraint representing the delivery timeliness requirement. and The weighting coefficient is used to balance the impact of priority and timeliness. It is the attenuation coefficient; score The highest-ranking queue type is selected to carry the data fragment. Routing nodes in the data distribution network continuously probe the network path status to the target data warehouse. These nodes assess the network path status by actively sending probe packets or passively analyzing past traffic. Network path status evaluation metrics include latency, packet loss rate, and available bandwidth. Routing nodes update this real-time path status information to a shared routing database. Combining the service level of the transmission queue with the real-time probed network path status, routing nodes dynamically select the optimal next-hop node or transmission link for each data fragment. The selection process involves a comprehensive evaluation of the latency, packet loss rate, bandwidth, and cost of multiple candidate paths. The selection algorithm tends to choose the most stable and lowest-latency path for high-priority or time-sensitive data fragments.

[0062] Optionally, if network congestion or path failure is detected during transmission, the routing node can perform rerouting or temporary storage operations according to a preset policy. The preset policy defines the actions to be taken when the performance indicators of a specific path deteriorate beyond a threshold. The routing node can reroute data fragments from the currently congested path to a better-performing backup path. Rerouting involves updating the next-hop information in the data fragment header. If all available paths are unavailable, the routing node can temporarily store the data fragments in local persistent storage to wait for network recovery and continue transmission after recovery.

[0063] When a data fragment successfully arrives at the receiving gateway of the target data warehouse, the receiving gateway verifies the integrity of the data fragment and sends an acknowledgment receipt. The integrity verification is performed by comparing the checksum or hash value of the data fragment. The acknowledgment receipt is returned to the sending routing node or distribution routing module along the original path or control path to confirm successful delivery. After the data fragment that has successfully passed the verification is unpacked and reassembled by the receiving gateway, it is stored in the designated storage location of the data warehouse. The designated storage location may correspond to a specific database table, data partition, or file system directory. This location information is ultimately determined by the routing label of the data fragment or the embedded metadata.

[0064] See Figure 5 This is a heatmap showing the fit between data probes and acquisition tasks. The color depth visually reflects the degree of fit between different probes and specific tasks. Probe 4 has the highest fit score for task 4 (close to 90), indicating that it is highly compatible with the requirements of task 4 in terms of computing power, network bandwidth, and geographical proximity, making it the optimal choice for this task. Transforming the abstract fit calculation into an intuitive heatmap provides a clear decision-making reference for task allocation algorithms. By identifying highly compatible combinations, resource waste can be reduced, and the overall operating efficiency of the acquisition system can be improved. This chart can serve as a benchmark for subsequent algorithm iterations to verify the effectiveness of the task allocation strategy. As a visualization tool for the dynamic acquisition logic framework, it can reflect changes in probe-task fit in real time, supporting the system to quickly adjust its strategy when data sources change dynamically.

[0065] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A normalized data acquisition system based on big data, characterized in that, The system includes: The probe sensing module deploys multiple data probes in a distributed network environment. These multiple data probes monitor and sense data sources with different protocols and structures in real time, thereby forming a raw sensing data stream. The portrait generation module parses and captures features of the original perceptual data stream, extracts multi-dimensional feature information that represents the inherent characteristics of the data source, and generates a dynamic portrait of the data source based on the multi-dimensional feature information. The data acquisition scheduling module establishes a dynamically adjusted data acquisition logic framework based on the dynamic profile of the data source. This data acquisition logic framework is used to guide the collaborative acquisition behavior of the data probes. Under the guidance of the data acquisition logic framework, the data conversion module performs synchronous or asynchronous acquisition actions by multiple data probes, sending the acquired raw data into a unified data conversion pipeline. The unified data conversion pipeline performs pattern recognition and structural reshaping on the raw data, converting the heterogeneous raw data into a data set with a unified pattern specification. The distribution and routing module injects the data set with a unified pattern specification into a data distribution network with buffering and routing functions, and the data distribution network then delivers the data to the designated data warehouse.

2. The normalized data acquisition system based on big data according to claim 1, characterized in that, The process of parsing and capturing features from the original sensory data stream, extracting multi-dimensional feature information representing the intrinsic characteristics of the data source, and generating a dynamic profile of the data source based on the multi-dimensional feature information specifically includes: The header and payload information of the data packets are separated from the original sensing data stream. The header information is decoded according to the protocol, and the payload information is analyzed by shallow syntax analysis to obtain the basic descriptor of the data stream. The basic descriptor is processed by time serialization to construct a time series signal describing the evolution of data arrival interval, data packet size change, and data content entropy over time; Signal processing methods are used to filter the time series signal to suppress noise, and periodic patterns, trend components, and abrupt change information in the time series signal are detected. The detected periodic patterns, trend components, and mutation point information are fused to construct a composite feature vector that can reflect the activity, stability, and suddenness of the data source. Simultaneously, the data generation context implicit in the original sensing data stream is analyzed, which includes the physical deployment location of the data source, logical network topology, and interaction traces with the application layer. The composite feature vector is associated and mapped with the data generation context to form a data source description model that includes static attributes and dynamic behavioral features. The data source description model is the dynamic profile of the data source.

3. The normalized data acquisition system based on big data according to claim 2, characterized in that, The step of using signal processing methods to filter the time series signal to suppress noise and to detect periodic patterns, trend components, and abrupt change information in the time series signal specifically includes: An adaptive filter is used to smooth the time series signal, and the parameters of the adaptive filter are automatically adjusted according to the local statistical characteristics of the time series signal. Spectral analysis is performed on smoothed time series signals to identify hidden periodic patterns by recognizing significant peaks in the spectrum. Meanwhile, the trend decomposition algorithm is used to separate the time series signal into long-term trend component, seasonal component and residual component; A mutation detection algorithm is applied to the residual components to identify short-term, drastic change points in the signal, which correspond to sudden events in the data source. The identified periodic pattern parameters, the mathematical expressions of the long-term trend components and seasonal components obtained by the trend decomposition algorithm, and the location and intensity information of the abrupt change points are recorded in a structured manner.

4. The normalized data acquisition system based on big data according to claim 2, characterized in that, The step involves establishing a dynamically adjustable data acquisition logic framework based on the dynamic profile of the data source. This framework guides the collaborative acquisition behavior of the data probes and specifically includes: The data source description model contained in the dynamic profile of the data source is analyzed, and quantitative indicators about the expected lifespan of the data source, the estimated data value density, and the urgency of data collection are extracted from it. Based on the aforementioned quantitative indicators, a multi-objective optimization function is designed. This multi-objective optimization function aims to balance the integrity and timeliness of data collection with the load pressure on the network and the data source itself. Solving the multi-objective optimization function yields a set of optimal acquisition strategy parameters for dynamic profiling of different data sources. These optimal acquisition strategy parameters include acquisition frequency, acquisition depth, and acquisition parallelism. The optimal acquisition strategy parameters are matched with the real-time status information of the data probes, which includes processing capacity margin, network bandwidth usage, and current task queue length. Based on the matching results, a specific acquisition instruction set is assigned to each data probe, and the communication protocol and data format for task collaboration and status synchronization between data probes are specified. All data probes assigned to a set of acquisition instructions form a logically unified but physically distributed acquisition cluster under the constraints of the communication protocol and data format. The overall behavior rules of the acquisition cluster constitute the dynamically adjusted data acquisition logic framework.

5. The normalized data acquisition system based on big data according to claim 4, characterized in that, The optimal acquisition strategy parameters are matched with the real-time status information of the data probes, including processing capacity margin, network bandwidth usage, and current task queue length. Based on the matching results, a specific acquisition instruction set is assigned to each data probe, specifically including: Establish a resource matching matrix, where the rows of the resource matching matrix represent data probes and the columns represent the dimensions of the collection task requirements defined by the optimal collection strategy parameters; The real-time status information of each data probe is quantified into a resource supply vector, and the optimal acquisition strategy parameters of each acquisition task are quantified into a resource demand vector. Calculate the fit score between each resource supply vector and all resource demand vectors. The fit score takes into account the capacity matching degree, load balancing degree and geographical proximity. A task allocation algorithm is used to assign collection tasks to the most suitable data probes based on the suitability score, thus forming a preliminary task allocation scheme. The degree to which the preliminary task assignment scheme achieves the overall goal of the data acquisition logic framework is evaluated. If the degree of achievement is lower than the threshold, the parameters of the task assignment algorithm are adjusted or constraints are added for iterative optimization until the degree of achievement meets the requirements. The finalized task assignment scheme is compiled into a set of acquisition instructions that can be executed by each data probe. The acquisition instruction set includes the target data source identifier, acquisition protocol, parameter configuration, and task scheduling plan.

6. The normalized data acquisition system based on big data according to claim 5, characterized in that, The task allocation algorithm is adopted to assign data collection tasks to the most suitable data probes based on the suitability score, forming a preliminary task allocation scheme, which specifically includes: The fit scores between the resource supply vector of each data probe and the resource demand vector of each acquisition task are arranged into a fit score matrix. In the fit score matrix, the data probe with the highest fit score for each acquisition task is selected as the candidate execution probe for the acquisition task. Check the number of acquisition tasks assigned to each candidate execution probe. If the number of acquisition tasks assigned to a candidate execution probe exceeds the upper limit that its processing capacity margin can bear, then mark the acquisition task with the lowest fit score on the candidate execution probe as a task to be reassigned. For each task to be reassigned, find the next data probe with the highest fit score. If the data probe is not currently assigned a task or the number of assigned tasks has not reached the upper limit, then assign the task to be reassigned to the data probe. Repeat the checking and redistribution process until all acquisition tasks are assigned to a single data probe, and the number of acquisition tasks assigned to each data probe does not exceed its processing capacity margin limit. At this point, a preliminary task allocation scheme is obtained.

7. The normalized data acquisition system based on big data according to claim 4, characterized in that, Under the guidance of the data acquisition logic framework, multiple data probes perform synchronous or asynchronous acquisition actions, sending the acquired raw data into a unified data transformation pipeline, specifically including: Each data probe establishes a connection session with the target data source according to its assigned set of acquisition instructions, and performs data extraction operations according to the specified acquisition frequency, acquisition depth and acquisition parallelism. During the data extraction process, the data probe continuously monitors the session status and data stream quality, and feeds back the monitored session status and data stream quality information to the dynamically adjusted data acquisition logic framework in real time. The dynamically adjusted data acquisition logic framework dynamically adjusts the acquisition instruction set of relevant data probes based on the feedback session status and data stream quality information, forming a closed-loop control. The data probe caches the successfully extracted raw data blocks locally and attaches metadata tags, which record the source, collection time, session identifier, and data format fingerprint of the data blocks. When the locally cached data block reaches a certain threshold or a transmission instruction is received, the data probe encapsulates the data block with metadata tags into a transmission unit according to the communication protocol and data format. The transmission units generated by multiple data probes are pushed in an orderly manner to the entry buffer of the unified data conversion pipeline through an asynchronous message mechanism or a synchronous call interface.

8. The normalized data acquisition system based on big data according to claim 7, characterized in that, The unified data transformation pipeline performs pattern recognition and structural reshaping on the raw data, transforming heterogeneous raw data into a data set with a unified pattern specification, specifically including: The unified data conversion pipeline's inlet buffer receives transmission units from multiple data probes, decapsulates the transmission units, and separates the original data blocks and their accompanying metadata tags. Based on the data format fingerprint in the metadata tag, a pre-built or dynamically loaded format parser is invoked to perform deep parsing of the original data block, identifying the internal structure, fields, types and constraints of the data; The identified data's internal structure, fields, types, and constraints are matched and mapped to a globally unified schema definition library, which defines the target specification's data model. Based on the matching and mapping results, the data in the original data block is cleaned, transcoded, split, merged, or the derived fields are calculated to eliminate ambiguity, fill missing information, correct errors, and meet the constraints of the target data model. After the data has been cleaned, transcoded, split, merged or derived fields have been calculated, it is reorganized into new data records according to the target data model, and a globally unique identifier and version stamp are generated for each new data record. All new data records that conform to the target data model are batched into data blocks, which together constitute a data set with a unified schema specification.

9. The normalized data acquisition system based on big data according to claim 8, characterized in that, The operations of cleaning, transcoding, splitting, merging, or calculating derived fields of the data in the original data block based on the matching and mapping results specifically include: For data with missing fields or abnormal value ranges, data is filled in according to the default value rules defined in the globally unified pattern definition library or the filling model learned from historical data. For text data with inconsistent encoding formats, a conversion table or character set detection technology is used to uniformly convert them into the specified standard encoding. For complex data that is too deeply nested or has an overly complex structure, it should be split into multiple related simple data records in accordance with the flattening requirements of the target data model. For data that logically belong to the same entity but are physically scattered in different original data blocks, they are merged according to entity parsing rules to generate a complete entity record; Based on business rules or statistical analysis requirements, new derived fields are generated by using the basic fields in the original data block through predefined functions or scripts. All operations involving cleaning, transcoding, splitting, merging, or calculating derived fields are recorded in a data tracing log, which is associated with the final generated new data record.

10. The normalized data acquisition system based on big data according to claim 9, characterized in that, The step of injecting the data set with a unified schema specification into a data distribution network with buffering and routing functions, and then having the data distribution network ultimately deliver the data to the designated data warehouse, specifically includes: The data set with a unified pattern specification is divided into appropriately sized data fragments, and each data fragment is attached with a routing label containing the target data warehouse address, data priority, and delivery time requirements; Based on the data priority and delivery time requirements in the routing labels, data fragments are allocated to transmission queues of different service levels in the data distribution network; Routing nodes in the data distribution network continuously probe the network path status leading to the target data warehouse, and the network path status includes latency, packet loss rate, and available bandwidth. Combining the service level of the transmission queue with the real-time detected network path status, the routing node dynamically selects the optimal next-hop node or transmission link for each data fragment. During transmission, if network congestion or path failure is detected, the routing node can re-route the data fragments to an alternative path according to a preset strategy, or temporarily store them in local persistent storage to await recovery. When a data fragment successfully arrives at the receiving gateway of the target data warehouse, the receiving gateway verifies the integrity of the data fragment and sends an acknowledgment. After being unpacked and reassembled, the data fragment is stored in the designated storage location of the data warehouse.

Images