Data weaving device supporting cross-domain data fusion analysis

Abstract

The invention provides a data weaving device supporting cross-domain data fusion analysis, and the device comprises a cross-domain data access storage module which is used for carrying out the distributed asynchronous intake processing of an original cross-domain data set, so as to generate an original data storage pool; the multi-dimensional data preprocessing module is used for performing multi-dimensional data cleaning and standardization processing on the cross-domain data in the original data storage pool to generate a standardized cross-domain data set; the cross-domain data association fusion module is used for performing cross-domain association fusion processing on the standardized cross-domain data set to generate a cross-domain fusion data set; the structured storage module is used for performing structured storage and analysis modeling processing on the cross-domain fusion data set to generate a cross-domain data fusion analysis result; and the fusion analysis result output packaging module is used for performing visual output and interface packaging processing on the cross-domain data fusion analysis result so as to generate an interactive cross-domain data fusion analysis output result. The device adapts to efficient processing requirements of cross-domain data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, in particular to a data weaving device supporting cross-domain data fusion analysis. BACKGROUND

[0002] In the intelligent development process of the fields of agriculture, industry, etc., the scale of multi-source cross-domain data (such as bag files collected by agricultural robots, real-time data of environmental sensors, image data and associated metadata generated by image collection devices) continues to expand. Enterprises need to integrate and analyze these cross-domain data to support core business scenario requirements, such as optimizing robot crop recognition algorithms in the agricultural scenario to improve picking accuracy, and implementing operation and maintenance status monitoring based on device multi-source data in the industrial scenario. Efficient processing and value mining of such cross-domain data have become the key demand of the development of the industry technology.

[0003] In the prior art, the implementation idea of "single storage architecture + artificial dominant processing" is often used for cross-domain data processing. First, cross-domain data of different sources is manually uploaded to a single storage system (such as a local server or a basic cloud storage), and then a customized script is written by a technical personnel to correct and simply clean the data. Finally, a separate data analysis tool is used to locally select and count the stored data to generate a basic data report.

[0004] In the above-mentioned prior art, the access, cleaning and correlation analysis of cross-domain data rely on manual operation and customized scripts, which not only leads to low data processing efficiency (such as several weeks are needed for a single batch of cross-domain data from access to the generation of analysis results), but also introduces problems such as non-uniform data format and inaccurate correlation logic due to manual intervention. At the same time, it is difficult to realize the deep correlation and fusion of data of different sources, and it is difficult to meet the technical needs of enterprises for fast and accurate analysis of cross-domain data to support business decisions. SUMMARY

[0005] To solve the above technical problems, the present application provides a data weaving device supporting cross-domain data fusion analysis to at least alleviate the above technical problems.

[0006] The technical scheme provided by the embodiments of the present application is as follows: A data weaving device supporting cross-domain data fusion analysis includes: a cross-domain data access and storage module for performing distributed asynchronous ingestion processing on the original cross-domain data set to generate an original data storage pool; a multi-dimensional data preprocessing module for performing multi-dimensional data cleaning and standardization processing on the cross-domain data in the original data storage pool to generate a standardized cross-domain dataset; a cross-domain data association and fusion module for performing cross-domain association and fusion processing on the standardized cross-domain dataset to generate a cross-domain fused dataset; a structured storage module for performing structured storage and analytical modeling processing on the cross-domain fused dataset to generate cross-domain data fusion analysis results; and a fusion analysis result output and encapsulation module for performing visualization output and interface encapsulation processing on the cross-domain data fusion analysis results to generate interactive cross-domain data fusion analysis output results.

[0007] The technical solution in this application has the following technical advantages: 1. Regarding the issue of "low access efficiency caused by manual uploading to a single storage device": The cross-domain data access and storage module of this device processes raw cross-domain data sets through "distributed asynchronous ingestion," eliminating the need for manual uploading. It can simultaneously process data from multiple sources (such as bag files from agricultural robots and sensor data) and generate a centralized raw data storage pool. Compared to traditional manual single-storage access methods, manual intervention in the data access process is significantly reduced, resulting in higher access efficiency and alleviating the problem of excessively long processing times for single-batch data access.

[0008] 2. Regarding the issue of "inconsistent formatting caused by simple manual script cleaning": The multi-dimensional data preprocessing module of this device is specifically designed to perform "multi-dimensional cleaning and standardization" on cross-domain data in the original data storage pool. It can cover unified optimization of multiple dimensions such as data format and field structure, and finally generate a standardized cross-domain dataset. Compared with traditional simple cleaning, the cross-domain data processed by this module has better format uniformity, which can alleviate the format chaos caused by manual intervention.

[0009] 3. Regarding the issue of "difficulty in achieving deep correlation and fusion of data from different sources": The cross-domain data association and fusion module of this device is specifically designed for "cross-domain association and fusion processing" of standardized cross-domain datasets. It can establish targeted association logic for data from different sources to generate cross-domain fused datasets. Compared with traditional partial filtering, this module achieves a deeper level of cross-domain data association and fusion, which can alleviate the problem of the inability to deeply integrate data from different sources.

[0010] 4. Regarding the issue of "inability to provide rapid and accurate analysis to support business decisions": The structured storage module of the device performs "structured storage and analysis modeling processing" on the cross-domain fusion data set, can orderly store the fused data and generate cross-domain data fusion analysis results through modeling; and in combination with the fusion analysis result output encapsulation module, generates "interactive cross-domain data fusion analysis output results". Compared with traditional basic reports, the structured degree and interactivity of the analysis results are relatively stronger, which can better support business decision-making needs and alleviate the problem of insufficient practicality of traditional analysis results. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 FIG. 1 is a structural schematic diagram of a data weaving device supporting cross-domain data fusion analysis according to an embodiment of the present application.

[0012] Figure 2 FIG. 2 is a flowchart of a data weaving method supporting cross-domain data fusion analysis according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] Figure 1 FIG. 1 is a structural schematic diagram of a data weaving device supporting cross-domain data fusion analysis according to an embodiment of the present application. As shown in FIG. 1, a data weaving device supporting cross-domain data fusion analysis includes: a cross-domain data access storage module, configured to perform distributed asynchronous ingestion processing on a raw cross-domain data set to generate a raw data storage pool; a multi-dimensional data preprocessing module, configured to perform multi-dimensional data cleaning and standardization processing on cross-domain data in the raw data storage pool to generate a standardized cross-domain data set; a cross-domain data association fusion module, configured to perform cross-domain association fusion processing on the standardized cross-domain data set to generate a cross-domain fusion data set; a structured storage module, configured to perform structured storage and analysis modeling processing on the cross-domain fusion data set to generate a cross-domain data fusion analysis result; and a fusion analysis result output encapsulation module, configured to perform visual output and interface encapsulation processing on the cross-domain data fusion analysis result to generate interactive cross-domain data fusion analysis output results. Figure 1 Optionally, the cross-domain data access storage module specifically includes the following three sub-modules: a multi-source data protocol adaptation access sub-module, configured to perform multi-protocol asynchronous access processing on the raw cross-domain data set, adapt communication protocols of different data sources to realize real-time / batch asynchronous access of data, and generate a cross-domain data stream accessed initially; a cross-domain data verification sub-module, configured to perform format legality verification and temporary storage processing on the cross-domain data stream to generate a temporary cross-domain data set in line with the format; and a cross-domain data hierarchical cloud storage sub-module, configured to perform classification storage scheduling processing on the temporary cross-domain data set in line with the format, store the data based on data types to a hierarchical structure directory, and generate the raw data storage pool.

[0014]

[0015] ​Optionally, the multi-source data protocol adaptation access submodule specifically comprises the following three components: a data source type identification component for performing data identification and analysis processing on the original cross-domain data set to generate a data source type classification result; a protocol adaptation configuration generation component for matching the original cross-domain data set with a preset protocol library based on the data source type classification result to generate protocol adaptation configuration information; and an asynchronous data access execution component for performing real-time / batch asynchronous transmission processing on the original cross-domain data set based on the protocol adaptation configuration information to generate a preliminarily accessed cross-domain data stream.

[0016] Preferably, the data source type identification component is specifically implemented as follows: first, taking the original cross-domain data set as the processing object, the packet header of each data unit is extracted by a data packet header parser (such as the "Device-Type" field of sensor data, the "PLC-Model" field of numerical control machine tool data, and the "Cam-SN" field of image device data), and initial identification information (such as "Device-Type:TemperatureSensor" and "PLC-Model:S7-1200") is obtained; then, taking the initial identification information as the processing object, a preset dynamic identification library (which contains the identification features of common data source types and the extension interface of new types) is called, and the initial identification information is compared with the features in the library (such as "TemperatureSensor" corresponding to "temperature and humidity sensor type" and "S7-1200" corresponding to "Siemens PLC type") by a feature matcher to obtain a preliminary type matching result; then, taking the preliminary type matching result as the processing object, the structure features of the data payload (such as "time + value" key-value pairs for sensor data, "register address + instruction code" for PLC data, and "resolution + frame rate" for image metadata) are combined for secondary verification to correct the deviation that may occur in single identification matching (such as avoiding misjudgment of "same name but different type" devices), and a data source type classification result (such as "Type 1: temperature and humidity sensor; Type 2: Siemens PLC; Type 3: industrial camera") is generated. This technology is different from the traditional identification method based on a fixed identification list. Through the combination of the dynamic identification library and the secondary verification of the payload structure, it can adapt to new data source types (such as not needing to reconfigure the identification logic when a new Internet of Things sensor is accessed), and the expansibility of cross-domain data source type identification is improved.

[0017] Preferably, the specific implementation process of the protocol adaptation configuration generation component is as follows: first, taking the data source type classification result as the processing object, querying the preset protocol library (which contains the mainstream communication protocols and parameter templates corresponding to various types of data sources, such as MQTT (Message Queue Telemetry Transport) for sensors, Modbus (a serial communication protocol) for PLCs, and HTTP (Hypertext Transfer Protocol) for industrial cameras), and obtaining the protocol type matching result (such as "temperature and humidity sensor → MQTT protocol; Siemens PLC → Modbus protocol"); then taking the protocol type matching result as the processing object, extracting the core configuration parameters of each protocol (such as "Broker address, client ID, and subscription topic" for MQTT protocol, and "serial port baud rate, slave address, and register start address" for Modbus protocol), and combining the scene characteristics of the data source type (such as the need for higher transmission priority for PLC data in industrial scenarios, corresponding to the "timeout time" of the Modbus protocol being set to a lower value), to generate an initial protocol parameter set; then taking the initial protocol parameter set as the processing object, checking the logical consistency between the parameters through the protocol compatibility detector (such as the need for "QoS (Quality of Service) level" and "retransmission times" of MQTT to match, to avoid the contradiction of "QoS = 2" but setting "retransmission times = 0"), and optimizing the parameters based on the optimal configuration in the historical access log (such as the higher transmission stability of a certain type of sensor under the "heartbeat interval = 30 seconds" of the MQTT protocol), to obtain the protocol adaptation configuration information (including protocol type, complete parameter list, and scene-based adaptation instructions). This technology is different from the traditional static protocol configuration method, and through scene-based parameter optimization and compatibility checking, it solves the protocol parameter conflict problem of different data sources in unified access, and improves the scene adaptability of data transmission.

[0018] Preferably, in the specific technical implementation of the asynchronous data access execution component: first, take the protocol adaptation configuration information and the original cross-domain data set as the common processing object, analyze the characteristics of the original cross-domain data set through the transmission mode determinator (such as sensor data is generated in high frequency and real-time, which is determined as "real-time transmission mode"; the log file of the numerical control machine tool is periodically generated in batches, which is determined as "batch transmission mode"), and obtain the transmission mode division result; then, take the transmission mode division result and the protocol adaptation configuration information as the processing object, assign the stream processing channel based on Kafka (a high-throughput distributed publish-subscribe message system) for the real-time transmission mode (each data source corresponds to an independent Topic (topic), which ensures data isolation), assign the batch processing task queue based on FTP (file transfer protocol) for the batch transmission mode (sort by file size, preferentially transmit files smaller than the preset threshold), and generate the transmission channel configuration result; then, take the transmission channel configuration result as the processing object, call the asynchronous transmission interface of the corresponding protocol (such as calling the asynchronous publish (publish) method of the Paho-MQTT client for the MQTT protocol, and calling the asynchronous reading function of the libmodbus library for the Modbus protocol), push the original cross-domain data set to the corresponding channel according to the transmission mode, and record the transmission state of each data ("to be transmitted", "transmitting", "transmission completed") to obtain the transmission state tracking table; finally, take the output data of the transmission channel and the transmission state tracking table as the processing object, sort the output data of different channels according to the timestamp through the data aggregator (take the collection timestamp for real-time data, and take the transmission start timestamp for batch data), eliminate the transmission failed data (filter the "transmission completed" items based on the transmission state tracking table), integrate to form a continuous data stream, and generate the cross-domain data stream accessed initially. This technology is different from the traditional single synchronous transmission mode. Through real-time / batch dual-mode asynchronous processing and state tracking, it not only meets the low delay demand of high-frequency real-time data, but also avoids the blocking of batch large data to the real-time channel, and improves the overall efficiency and reliability of cross-domain data access.

[0019] Optionally, the cross-domain data verification sub-module specifically includes the following three components: a verification rule dynamic generation component for generating a data verification rule set according to the data attributes in the cross-domain data stream; a multi-dimensional verification execution component for performing data item-by-data item verification processing on the cross-domain data stream and the data verification rule set to generate compliant data segments; and a compliant data distributed cache component for performing distributed temporary storage processing on the compliant data segments and establishing a data index to generate a temporary cross-domain data set in a format compliant with the regulations.

[0020] Preferably, the specific implementation process of the verification rule dynamic generation component is as follows: first, taking cross-domain data flow as the processing object, the inherent attributes of each data item in the cross-domain data flow are extracted through the field parser, including data type (such as "sensor numerical value type" and "image metadata character type" in the agricultural scene), field length (such as device ID fixed 16-bit character), and associated field identification (such as the binding relationship between "collection time" and "plot number"), to obtain a data attribute list; then, taking the data attribute list as the processing object, combining the business requirements of the target application scene (such as the agricultural cross-domain data needing to meet "collection time non-empty" and "humidity value needs to match the plot type"), the format verification dimension, the value range verification dimension, and the associated logic verification dimension are divided, and the verification dimension division result is generated; then, taking the verification dimension division result as the processing object, the preset rule template library is matched for each dimension (such as the format verification dimension matching "time format template" and "character encoding template", and the value range verification dimension matching "numerical interval template"), and the scene parameter threshold is imported (such as the conventional interval of the agricultural greenhouse humidity value being 40%-80%), to generate the initial verification rule; thereafter, taking the initial verification rule as the processing object, the logical contradiction between different dimension rules is compared through the rule conflict detector (such as the conflict between "numerical value allowed to be empty" and "associated logic required to be non-empty"), and the conflict rule is corrected based on the scene priority (such as the "collection time non-empty" priority being higher than the "remark field non-empty" priority in the agricultural scene), to obtain the data verification rule set. This technology is different from the traditional fixed rule generation method, and through the dynamic adaptation of the attributes of the cross-domain data flow and the scene requirements, the "one-size-fits-all" problem of traditional rules for different source data is avoided, so that the generated verification rule has higher scene adaptability.

[0021] Preferably, the multi-dimensional verification execution component is implemented as follows: first, taking cross-domain data streams and data verification rule sets as common processing objects, the data shuffler splits the cross-domain data streams according to "data source type + collection time slice" to obtain data slices grouped by scenarios (such as "2024-10-15 08:00-09:00 greenhouse sensor data slice" and "2024-10-15 09:00-10:00 unmanned aerial vehicle bag file data slice"); then, taking the data slices grouped by scenarios as processing objects, a corresponding multi-dimensional verification thread (format verification thread, value range verification thread, and correlation logic verification thread) is allocated to each slice, and the corresponding dimension rules in the data verification rule set are called to verify each data item in the slice one by one (for example, the format verification thread verifies whether the "collection time" conforms to the "YYYY-MM-DD HH:MM:SS" format, and the value range verification thread verifies whether the "soil nitrogen content" is within the interval of 0.1-5.0 g / kg); a single-dimensional verification result list is obtained (each result contains a data item identifier, a dimension verification result, and an illegal reason); then, taking the single-dimensional verification result list as a processing object, the result aggregator determines whether each data item meets the rules of all verification dimensions (format, value range, and correlation logic verification); filters out data items that meet all dimensions, and combines them in the original data slice time sequence to generate a compliant data segment. This technology is different from the traditional single-dimensional serial verification method. Through parallel processing and multi-dimensional synchronous verification, the overall efficiency of cross-domain data verification can be improved, and the multi-source data characteristics of the agricultural scene are grouped for verification, reducing the verification interference between different types of data.

[0022] Preferably, in the specific technical implementation of the compliance data distributed cache component: first, take the compliance data segment as the processing object, extract the core identification features of each segment (such as "data source device ID", "data collection plot code", "data type identification" in the agricultural scene) through the feature extractor, and obtain the segment feature information (such as "device ID: SN2024AG001; plot code: A01; data type: soil temperature and humidity"); then take the segment feature information as the processing object, use the consistent hash algorithm to calculate the cache node hash value corresponding to each segment (the input parameter of the hash calculation is "device ID+plot code", and the output is an integer between 0 and 65535, corresponding to the node number of the distributed cache cluster), and obtain the cache node allocation result (such as "segment 1→node 3, segment 2→node 5"); then take the compliance data segment and the cache node allocation result as the common processing object, transmit each segment to the corresponding cache node through the data transmission protocol (such as the TCP protocol), call the write interface of the node to perform the data persistence operation (use the LSM tree storage structure to support high write throughput), and obtain the cache write confirmation information (including the segment storage address and the write timestamp); at the same time, take the segment feature information and the cache write confirmation information as the processing object, build a distributed data index (the index key is "device ID+collection time range", and the index value is the cache node address+segment storage address), and synchronize to the index center node of the cache cluster; finally, take all the compliance data segments of the cache nodes and the distributed data index as the processing object, map the segments and the index through the index associator, integrate to form a unified data set management view, and generate a format-compliant temporary cross-domain data set. This technology is different from the traditional random distributed cache method. Through the feature-based node allocation and index construction, the positioning efficiency during subsequent data reading can be improved, and the LSM tree storage structure is suitable for the high-write scenario of cross-domain data (such as the continuous access of hundreds of sensor data per second in the agricultural scene), avoiding the write performance bottleneck of the traditional B+ tree storage.

[0023] Optionally, the cross-domain data hierarchical cloud storage submodule specifically includes the following three components: a storage data type subdivision component: used for performing data structure analysis processing on the format-compliant temporary cross-domain data set to generate data type subdivision results; a hierarchical directory mapping component: used for generating a correspondence between the data type and the cloud storage hierarchical directory according to the data type subdivision results, and then generating a directory mapping relationship table; a storage scheduling execution component: used for performing classified storage scheduling processing on the format-compliant temporary cross-domain data set based on the directory mapping relationship table, and writing different types of data into the corresponding hierarchical directory, to generate an original data storage pool.

[0024] Preferably, the storage data type subdivision component specifically implements: first, taking the format-compliant temporary cross-domain data set as the processing object, extracting the core structural features of each data unit through a data structure parser, including field composition (such as "collection time + plot ID + soil moisture value"), data form (such as numerical type, binary type, and text type), and data generation frequency (such as 1 piece per second, 1 piece per day), to obtain a data structure feature list; then, taking the data structure feature list as the processing object, combining the business attribute dimensions of smart agriculture (such as "environmental monitoring", "crop growth", and "equipment operation and maintenance"), and preliminarily classifying the data, for example, classifying "soil moisture value" and "air temperature value" as environmental monitoring, classifying "crop leaf image" and "fruit maturity photo" as crop growth, and classifying "farm machinery working time" and "fault code record" as equipment operation and maintenance, to generate a business attribute preliminary classification result; then, taking the business attribute preliminary classification result as the processing object, comparing the structural features of the data in the same category through a type consistency checker (such as environmental monitoring data needing to contain "timestamp + monitoring index value" fields, and crop growth data needing to have "image resolution + shooting position" meta information), eliminating classification deviation data (such as data that mistakenly classify farm machinery fault code text as environmental monitoring data), and reclassifying, to finally generate a data type subdivision result (such as "environmental monitoring - numerical data", "crop growth - binary image data", and "equipment operation and maintenance - text log data"). This technology is different from the traditional single classification method based on data format only. Through the dual verification of business attributes and structural features, the data classification is more suitable for subsequent storage and application scene requirements, avoiding the problem of low subsequent retrieval efficiency caused by "same format different purpose data mixed storage".

[0025] Preferably, the specific implementation process of the hierarchical directory mapping component is as follows: first, the data type subdivision result is taken as the processing object, a cloud storage hierarchical directory framework under the wisdom agriculture scenario is defined based on the hierarchical design principle of cloud storage (the hot data layer is used for high-frequency access data, the warm data layer is used for medium-frequency access data, and the cold data layer is used for low-frequency access data), the first-level directory of the framework is "data hierarchical identifier" (such as "Hot- hot data layer", "Warm- warm data layer", and "Cold- cold data layer"), the second-level directory is "business attribute category" (such as "environmental monitoring", "crop growth", and "device operation and maintenance"), and the third-level directory is "data type + time period" (such as "numerical data / 2024-10" and "image data / 2024-Q4"), and a hierarchical directory framework definition result is obtained; then, the hierarchical directory framework definition result and the data type subdivision result are taken as the common processing object, a preliminary mapping relationship is established, for example, "environmental monitoring- numerical data" (high-frequency access, used for real-time irrigation decision) is mapped to "Hot- hot data layer / environmental monitoring / numerical data / current month", "crop growth- binary image data" (medium-frequency access, used for weekly growth analysis) is mapped to "Warm- warm data layer / crop growth / image data / current quarter", and "device operation and maintenance- text log data" (low-frequency access, used for quarterly fault review) is mapped to "Cold- cold data layer / device operation and maintenance / text log / current year", and an initial directory mapping table is generated; then, the initial directory mapping table is taken as the processing object, the actual access frequency of each type of data is analyzed in combination with the historical data access log (for example, the daily average access number of "environmental monitoring- numerical data" in the past 3 months is 500 times, and the daily average access number of "device operation and maintenance- text log data" is 5 times), the mapping relationship is optimized and adjusted, for example, "crop growth- image data" with an increased access frequency (for example, the daily average access number increases from 20 times to 100 times after entering the crop harvesting period) is adjusted from the warm data layer to the hot data layer directory, and finally a directory mapping relationship table (containing data type, target hierarchical directory path, and access frequency adaptation specification) is generated. This technology is different from the traditional fixed directory mapping method, and the hierarchical mapping is dynamically optimized by combining the access frequency, which not only ensures the access efficiency of high-frequency data, but also reduces the storage cost of low-frequency data, and meets the use demand differences of different data under the wisdom agriculture scenario.

[0026] Preferably, in the specific technical implementation of the storage scheduling execution component: first, taking the directory mapping relationship table and the format-compliant temporary cross-domain data set as the common processing object, the format-compliant temporary cross-domain data set is split into multiple corresponding data shards (such as "environmental monitoring numerical value shard", "crop growth image shard", and "device operation and maintenance log shard") according to the data type classification in the directory mapping relationship table through a data sharding device, and each data shard carries a corresponding target hierarchical directory identifier, obtaining a classified data shard set; then taking the classified data shard set and the directory mapping relationship table as the processing object, an appropriate storage node is allocated for each classified data shard according to the storage characteristics of the target hierarchical directory (the hot data layer corresponds to the SSD storage node with high IO performance, the warm data layer corresponds to the SAS storage node with balanced performance, and the cold data layer corresponds to the SATA storage node with large capacity) through a storage node scheduler, for example, the "environmental monitoring numerical value shard" is allocated to the SSD node 1-3, the "crop growth image shard" is allocated to the SAS node 4-6, and the "device operation and maintenance log shard" is allocated to the SATA node 7-9, and a storage node allocation result is generated; then taking the classified data shard set and the storage node allocation result as the processing object, an asynchronous transmission interface of the corresponding storage protocol (the iSCSI protocol is used to realize low-latency transmission for the hot data layer, the NFS protocol is used to realize shared access for the warm data layer, and the S3 protocol is used to realize object storage for the cold data layer) is called to transmit each classified data shard to the target storage node and write it into the corresponding hierarchical directory, while recording the write state of each data shard ("to be written", "writing", "written successfully", and "written failed"), and obtaining a storage execution state table; finally, taking the storage execution state table and the directory mapping relationship table as the processing object, the classified data shards with "written successfully" are filtered out, associated with the target hierarchical directory path, integrated to form a data set management view containing all compliant storage data and the corresponding directory location, and a raw data storage pool is generated. This technology is different from the traditional single-node and single-protocol storage method. By allocating nodes and protocols according to hierarchical characteristics, the storage performance requirements of different data (such as low-latency access of hot data) are matched, the storage reliability of large-scale cross-domain data is guaranteed, and the problems of "performance excess" or "performance deficiency" in traditional storage are avoided.

[0027] Optionally, the multi-dimensional data preprocessing module specifically includes the following three sub-modules: a data anomaly detection processing submodule for identifying and removing anomalies in the cross-domain data in the raw data storage pool to generate normalized data after removing anomalies; a data format standardization submodule for uniformly processing the normalized data after removing anomalies to unify the field structure and data type of different source data into a preset standard format to generate standardized data with uniform format; and a data compression optimization submodule for compressing the standardized data with uniform format to generate a standardized cross-domain data set.

[0028] Optionally, the data anomaly detection processing submodule is configured to perform anomaly identification and elimination processing on the cross-domain data in the original data storage pool to generate normalized data after eliminating anomalies, and specifically includes the following two components: an anomaly detection rule dynamic generation component configured to analyze the data attributes and correlation of the cross-domain data in the original data storage pool to obtain standard features, so as to formulate dynamic anomaly judgment standards suitable for different hierarchical directory data; and a distributed anomaly data identification and elimination component configured to perform data item-by-item comparison processing on the cross-domain data in the original data storage pool according to the hierarchical directory and batch to determine and eliminate abnormal items, so as to generate normalized data after eliminating anomalies.

[0029] Preferably, when the anomaly detection rule dynamic generation component is implemented, first, the cross-domain data in the original data storage pool is taken as a processing object, the basic attributes (such as “the numerical type is floating point, the unit is ℃, and the collection frequency is 1 time / s” of motor temperature data, and “the numerical type is integer, the unit is Hz, and the collection frequency is 10 times / s” of vibration data) and the correlation (such as “when the motor temperature is greater than 80℃, the vibration frequency is usually in the range of 50-100Hz”, and “when the voltage fluctuation is greater than ±5%, the temperature rising rate will be accelerated”) of the data under each directory are extracted according to the hierarchical directory to obtain the hierarchical data attribute and correlation list; then, the hierarchical data attribute and correlation list is taken as a processing object, the same hierarchical directory data is statistically analyzed, the distribution characteristics (such as mean, standard deviation, and quartile value) of numerical data and the format characteristics (such as the fixed length of device ID and the coding rule of fault code) of text data are calculated, and single-dimensional standard features (such as “normal range of motor temperature: mean ± 3 times standard deviation” and “device ID format: 8-digit number + 2-letter”) are generated; then, the single-dimensional standard features and the correlation list are taken as processing objects, the physical logic of industrial equipment operation (such as “the temperature of the motor should be lower than that under load when the motor is under load” and “when the voltage is stable, the vibration frequency fluctuation range should be less than 10Hz”) is combined, and the correlation dimension standard features (such as “linkage threshold of temperature and vibration frequency: for every 10℃ increase in temperature, the vibration frequency is allowed to expand the fluctuation range by 5Hz”) are constructed; finally, the single-dimensional standard features and the correlation dimension standard features are taken as processing objects, the rule dynamic update trigger condition (such as “trigger update when the data proportion exceeding the current rule is greater than 5% in 1000 consecutive data” and “forced update when the device model is changed”) is set, and the dynamic anomaly judgment standards (divided according to the hierarchical directory, each directory contains single-dimensional rules, correlation dimension rules, and update mechanisms) are integrated. This technology is different from the traditional anomaly rule based on fixed threshold, which generates rules by combining the data distribution characteristics and the scene physical logic, and supports dynamic update, can adapt to the characteristics of cross-domain data changing with the equipment state and environment, and avoids the problem of insufficient adaptability of fixed rules to data drift.

[0030] Preferably, in the specific technical implementation of the distributed abnormal data identification and elimination component: first, the cross-domain data in the raw data storage pool and the dynamic abnormality judgment standard are taken as the common processing objects, the cross-domain data is processed in slices according to the hierarchical directory (for example, the "motor temperature data directory" is split into "M01_20241015" "M02_20241015" and other data slices according to "equipment number + date", and the "vibration data directory" is split according to the same rule), each slice contains cross-domain data of continuous time sequence, and a hierarchical sliced data set is obtained; then, the hierarchical sliced data set and the dynamic abnormality judgment standard are taken as the processing objects, a distributed computing framework (such as Apache Spark, a fast and general cluster computing system) is used to allocate independent computing nodes for each data slice, the nodes load the dynamic abnormality judgment standard of the corresponding hierarchical directory (such as the "M01 motor temperature" slice loads the "motor temperature hierarchical dynamic rule"), and a slice-node allocation result is generated; then, the slice-node allocation result is taken as the processing object, each computing node performs data item-by-item comparison on the data in the slice: first, single-dimensional features are verified (such as whether the motor temperature value is within the range of "mean value ± 3 times standard deviation", whether the equipment ID format conforms to the rule), and single-dimensional abnormal items are marked; then, the associated dimension features are verified (such as whether the vibration frequency is within the range of 55-105 Hz when the temperature is 85°C, whether the voltage fluctuation matches the temperature rise rate), and the associated dimension abnormal items are marked; the single-dimensional and associated dimension abnormal items are summarized, and a slice abnormal data marking table (containing the location, type and trigger rule of the abnormal data) is generated; then, the slice abnormal data marking tables of all computing nodes are taken as the processing objects, the abnormal markers of each slice are integrated through a result aggregator, the corresponding data is associated according to the directory structure of the raw data storage pool, and a global abnormal data list is obtained; finally, the cross-domain data in the raw data storage pool and the global abnormal data list are taken as the processing objects, a data elimination interface is called to remove the abnormal data marked in the list, the remaining data is retained and reorganized according to the hierarchical directory, and the standardized data after removing the abnormal data is generated. This technology is different from the traditional single-node serial detection method, and through hierarchical slicing and distributed parallel processing, the abnormality detection efficiency of large-scale cross-domain data can be greatly improved; at the same time, combined with the double comparison of single-dimensional and associated dimension, more hidden abnormalities (such as data with single-dimensional normal but associated logic contradiction) can be identified compared with single-dimensional detection, and the comprehensiveness of abnormality identification is improved.

[0031] Optionally, the data format standardization submodule specifically comprises the following three components: a standard format template versioning component for formulating a version-controllable standard format template for different types of data based on the data source type classification result; a microservice format conversion component for calling a special format conversion microservice deployed in a Kubernetes container based on the standard format template, performing distributed format conversion processing on the abnormality-removed data to be standardized to generate intermediate data after format conversion; and a consistency checking component for performing field structure comparison, data type checking, and label integrity checking on the intermediate data after format conversion to generate standardized data with uniform format.

[0032] Preferably, the standard format template versioning component is implemented as follows: first, classify the results by data source type, extract the core business fields of each type of data (for example, "image metadata type" needs to include "snap time, device ID, vehicle identification number, image resolution", "numerical monitoring type" needs to include "monitoring time, device ID, speed value, lane number", "device status type" needs to include "collection time, device ID, running status code, voltage value"), and determine the data type (for example, "snap time" is a string type in "YYYY-MM-DD HH:MM:SS.fff" format, "speed value" is a floating point type with 1 decimal place, "running status code" is an integer type with fixed 3 digits) and constraint rules (for example, "vehicle identification number" is fixed at 17 digits, "voltage value" is in the range of 180-240V) of each field to obtain a list of sub-type fields and constraints; then, using the list of sub-type fields and constraints as the processing object, define the unified structure framework of the template, including "basic information layer" (field name, data type, constraint rule), "association mapping layer" (cross-type data association fields, such as "device ID" in each type of data needs to be consistent in format), "scene extension layer" (reserved custom field interface, such as adding "event identifier" field when adding new traffic event type), and generate the standard format template basic structure; then, using the standard format template basic structure as the processing object, develop version control rules, determine the version number coding rules (for example, "V main version number. Sub version number", the main version number corresponds to the addition or deletion of fields, and the sub version number corresponds to the fine tuning of constraint rules), version update trigger conditions (such as adding new data source type, changing existing field constraints) and version compatibility mechanism (old version template can be converted to new version, keeping the core fields), and obtain the template version management rules; finally, using the standard format template basic structure and template version management rules as the processing object, generate the corresponding template file for each type of data source, and mark the version number and effective time (for example, "image metadata type-V1.0, 2024-10-01 effective" "numerical monitoring type-V1.0, 2024-10-01 effective"), and integrate to form a sub-type versioned standard format template. This technology is different from the traditional fixed template without version, and can adapt to data source type upgrade or business demand change through scenario-based field constraints and version management, avoiding the problem of new data not being able to adapt due to "one size fits all" in traditional templates.

[0033] Preferably, in the specific technical implementation of the micro-service format conversion component: first, take the standard format template of the division type versioning standard and the data to be standardized after the exception is removed as the common processing object, load the corresponding standard format template according to the data source type (match the "image metadata class" "numerical monitoring class" "device state class" in the template) of the data to be standardized, parse the field mapping rules in the template (such as the "Speed" field in the radar data is mapped to the "speed value" in the template, and the "CaptureTime" field in the camera data is mapped to the "snap time" in the template), and get the type-template-mapping rule association table; then take the type-template-mapping rule association table and the data to be standardized as the processing object, based on the resource scheduling capability of Kubernetes (K8s, a container orchestration platform), allocate dedicated format conversion micro-service instances (such as "image metadata conversion micro-service" deployed on GPU resource node, adapt to image resolution field format conversion; "numerical monitoring conversion micro-service" deployed on CPU resource node, adapt to speed value, voltage value precision conversion) for different data source types, and dynamically adjust the number of instances according to the data volume of the data to be standardized (such as single batch data volume > 100,000, automatically expand to 3 micro-service instances for parallel processing), generate micro-service scheduling configuration results; then take the micro-service scheduling configuration results as the processing object, and each micro-service instance performs format conversion according to the mapping rule: performs renaming on data with inconsistent field names (such as "DevID" in the signal machine data is converted to "device ID"), performs type conversion on data with unmatched data types (such as string type "speed value" is converted to floating point type), and performs format correction on data with non-standard format (such as "202410151230" is converted to "2024-10-15 12:30:00.000"), to get single-instance conversion results; finally, take the single-instance conversion results of all micro-service instances as the processing object, sort them by data collection timestamp through the result merger (ensure the time sequence consistency of cross-type data), integrate to form a continuous data stream, and generate the format converted intermediate data. This technology is different from the traditional monolithic format conversion. Through the dynamic scheduling and type-specific instance allocation of K8s containerized micro-service, it not only improves the conversion efficiency of large-scale cross-domain data, but also avoids the problem of overall interruption caused by single instance failure.

[0034] Preferably, the specific implementation process of the consistency checking component is as follows: first, taking the intermediate data after format conversion and the standard format template of the sub-type versioning as common processing objects, extracting the "basic information layer" constraints (number of fields, data type, field length) and "scene extension layer" label requirements (such as all data in the intelligent transportation scene must contain "lane ownership" and "device ownership road segment" labels) in the template, and generating a multi-dimensional checking reference; then taking the intermediate data after format conversion and the multi-dimensional checking reference as processing objects, performing field structure consistency checking: comparing whether the number of fields of the intermediate data is consistent with the number of template fields (such as "numeric monitoring type" data must contain 5 core fields, and if one is missing, it is determined to be inconsistent), checking whether the name of each field matches the template field name (no abbreviations or synonymous replacements are allowed, such as "vehicle speed" cannot be replaced by "speed value"), and obtaining a field structure checking result; then taking the intermediate data after format conversion and the multi-dimensional checking reference as processing objects, performing data type consistency checking: verifying whether the data type of each field conforms to the template definition (such as "running state code" must be a 3-digit integer, and if it is a string type or a 2-digit integer, it is determined to be inconsistent), and verifying whether the value range of the numeric type field meets the constraints (such as "voltage value" must be in the range of 180-240V, and if it exceeds, it is marked as abnormal), and obtaining a data type checking result; then taking the intermediate data after format conversion and the multi-dimensional checking reference as processing objects, performing label integrity checking: checking whether the intermediate data contains all the required scene labels (such as if the "device ownership road segment" label is missing in the roadside device data, it is determined to be incomplete), and verifying whether the format of the label value conforms to the convention (such as "lane ownership" must be in the format of "L + number", such as "L1" and "L2"), and obtaining a label integrity checking result; finally, taking the field structure checking result, the data type checking result, and the label integrity checking result as processing objects, retaining the data that passes all checking items, performing correction (renaming based on template mapping rules) on the data that is inconsistent in a single item (such as incorrect field name), marking invalid data and removing the data that is inconsistent in multiple items or cannot be corrected (such as missing core labels and cannot be completed), integrating the retained valid data, and generating standardized data with uniform format. This technology is different from the traditional single method of checking field format only. Through multi-dimensional checking of fields, types, and labels, it can ensure that the standardized data not only has uniform format, but also has complete scene association information, avoiding the "data island" problem caused by missing labels in subsequent cross-domain data association and fusion.

[0035] Optionally, the data compression optimization submodule specifically comprises the following three components: a compression algorithm intelligent matching component: configured to determine, based on the access frequency feature and the cost optimization principle, an optimal compression algorithm matching the data type of the format-unified standardized data from a preset compression algorithm library to generate a data-algorithm intelligent matching relationship table; a compression parameter dynamic configuration component: configured to configure dynamic compression parameters for different types of standardized data based on the data-algorithm intelligent matching relationship table and the data lake storage layering strategy to generate dynamic compression parameter configuration information; and an integrated compression component: configured to generate a "compression-checking-metadata recording" processing link based on the dynamic compression parameter configuration information, and perform compression processing on the format-unified standardized data based on the processing link to generate the standardized cross-domain data set.

[0036] Preferably, the compression algorithm intelligent matching component is implemented as follows: first, standardized data in a unified format is taken as the processing object, the basic types of data (such as "video frame sequence data", "floating-point numerical time series data", and "UTF-8 encoded text data") are distinguished through a feature extractor, and the access frequency features of each type of data (such as "traffic monitoring video data is accessed 10 times per day", "environmental sensor data is accessed once per hour", and "government text data is accessed twice per month") are extracted in combination with the access log of the original data storage pool to obtain a data type and access frequency feature table; then, the data type and access frequency feature table is taken as the processing object, a preset compression algorithm library (including video algorithms such as H.265 (high-efficiency video coding), numerical algorithms such as LZ77 (a lossless data compression algorithm), and text algorithms such as DEFLATE (a lossless data compression algorithm)) is called, and the core characteristics of each algorithm are labeled, including compression rate (volume ratio of compressed data to original data), decompression speed (unit data decompression time consumption), and calculation cost (CPU occupancy rate during compression process) to generate an algorithm characteristic list; then, the data type and access frequency feature table and the algorithm characteristic list are taken as the processing object, a matching model is established based on the cost optimization principle: for high-frequency access data (such as accessed ≥1 times per hour), algorithms with faster decompression speed (such as decompression time consumption <10 ms / MB) and lower calculation cost (such as numerical data matching LZ77 to avoid high-frequency decompression occupying too many resources) are preferred; for low-frequency access data (such as accessed ≤5 times per month), algorithms with higher compression rate (such as compression rate <30%) are preferred (such as government text data matching DEFLATE to reduce long-term storage occupancy) to generate a preliminary matching relationship; finally, the preliminary matching relationship is taken as the processing object, the adaptability of the matching algorithm to the data type is tested through an algorithm compatibility verifier (such as H.265 is not suitable for text data, and such incorrect matching needs to be excluded), and the data-algorithm intelligent matching relationship table (including data type, access frequency, matching algorithm, and adaptation reason) is obtained after correction. This technology is different from the traditional single matching method based only on data type. By combining access frequency and cost optimization principles, it not only guarantees the fast access demand of high-frequency data, but also reduces the long-term storage cost of low-frequency data, solving the problem of "compression rate and access efficiency imbalance" in traditional algorithm matching.

[0037] Preferably, the specific implementation process of the compression parameter dynamic configuration component is as follows: first, taking the data-algorithm intelligent matching relationship table and the data lake storage layering strategy (hot data layer corresponds to SSD storage, emphasizing access speed; warm data layer corresponds to SAS storage, balancing speed and cost; cold data layer corresponds to SATA storage, emphasizing storage capacity) as common processing objects, the configurable parameters of the matching algorithm (such as H.265 "quantization parameter QP" (affecting compression ratio and picture quality, the higher the QP value, the higher the compression ratio and the lower the picture quality), LZ77 "sliding window size" (affecting compression ratio and memory occupation, the larger the window, the higher the compression ratio and the larger the memory occupation), DEFLATE "compression level 0-9" (the higher the level, the higher the compression ratio and the longer the time consumption)) are parsed to obtain an algorithm parameter list; then, taking the algorithm parameter list and the storage layering strategy as processing objects, basic parameters are configured for different layered data: hot data layer data (such as high-frequency access traffic monitoring video) adopts lower compression intensity parameters (such as H.265 QP=25, ensuring decompression speed), warm data layer data (such as medium-frequency access environmental sensor data) adopts medium compression intensity parameters (such as LZ77 sliding window=32KB, balancing compression ratio and efficiency), and cold data layer data (such as low-frequency access government text) adopts higher compression intensity parameters (such as DEFLATE level=6, maximizing compression ratio), generating a layered basic parameter set; then, taking the layered basic parameter set and the standardized data of uniform format as processing objects, a dynamic adjustment factor is introduced: when the single batch data volume exceeds the preset threshold (such as >1GB), the compression intensity parameter of the cold data layer is reduced (such as DEFLATE from level 6 to 4, avoiding long compression time consumption); when the data contains key identification fields (such as "ID number" and "enterprise code" in government text), lossless compression parameters are forced to be used (such as LZ77 sliding window=64KB, ensuring that key information is not distorted), obtaining a dynamic adjustment parameter set; finally, taking the layered basic parameter set and the dynamic adjustment parameter set as processing objects, dynamic compression parameter configuration information containing parameter values, applicable scenarios, and adjustment trigger conditions is integrated (such as "traffic monitoring video (hot data layer): H.265, QP=25; data volume >1GB, QP=28"). This technology is different from the traditional fixed parameter configuration method, which dynamically adjusts the parameters by combining storage layering and data characteristics, avoiding the problems of "excessive compression leading to distortion" or "insufficient compression wasting space" of fixed parameters when the data volume fluctuates or contains key information.

[0038] Preferably, in the specific technical implementation of the integrated compression component: first, take the dynamic compression parameter configuration information and the data-algorithm intelligent matching relationship table as the common processing object, build the "compression-verification-metadata record" processing link framework: the compression link binds the matched algorithm and the dynamic parameter, the verification link sets the data integrity verification rule (such as generating the CRC32 (32-bit cyclic redundancy check) check value before and after compression for comparison), the metadata record link defines the information to be recorded (compression algorithm name, parameter value, data size before and after compression, compression timestamp, check value), and the link framework definition result is obtained; then, take the link framework definition result and the standardized data with uniform format as the processing object, start the corresponding compression process according to the data type: call the algorithm in the data-algorithm intelligent matching relationship table (such as video data calling H.265), load the parameter value in the dynamic compression parameter configuration information (such as QP=25), perform block compression on the standardized data (split large data volume data according to the preset block size (such as 100MB / block) and then perform parallel compression to improve efficiency), and obtain the block compressed data; then, take the block compressed data as the processing object, execute the verification link: calculate the CRC32 check value of each block of compressed data, compare it with the CRC32 check value of the corresponding data block before compression, if they are consistent, mark it as "verification passed", if they are inconsistent, re-execute the compression of the block (at most 3 times, still failed, mark it as "compression exception"), and obtain the compression verification result; thereafter, take the block compressed data, the compression verification result and the link framework definition result as the processing object, execute the metadata record link: generate the metadata entry for the "verification passed" block compressed data (such as "algorithm: H.265; parameter: QP=25; original size: 100MB; compressed size: 30MB; timestamp: 2024-10-15 14:30:00; check value: 0x12345678"), and form the compression metadata set by summarizing all the metadata entries; finally, take the block compressed data, the compression verification result and the compression metadata set as the processing object, integrate the "verification passed" block compressed data, associate the corresponding compression metadata set, reorganize according to the type of the original data and the hierarchical directory, and generate the standardized cross-domain data set (including the compressed data body and the traceable metadata). This technology is different from the traditional single process of only executing compression. Through the integrated link, the compression, verification and metadata record are bound, which not only guarantees the integrity of the compressed data (avoids data damage in the compression process), but also realizes the full traceability of the compression process through the metadata, solving the problems of "difficult to troubleshoot data damage" and "no record of compression parameters" in traditional compression.

[0039] Optionally, the cross-domain data association fusion module specifically comprises the following three sub-modules: a metadata association rule construction sub-module for parsing and association rule construction processing of the metadata of the standardized cross-domain data set, determining the association fields between different data source data, and generating a cross-domain data association rule set; a micro-service data fusion processing sub-module for fusing the standardized cross-domain data set by scheduling microservices with different functions through a container orchestration system based on the cross-domain data association rule set to generate a preliminary fusion data set; and a consistency checking sub-module for consistency verification processing of the preliminary fusion data set, checking and correcting field conflicts and logical contradictions of the fused data to generate a cross-domain fusion data set.

[0040] Optionally, the metadata association rule construction sub-module specifically comprises the following three components: a metadata field parsing and bloodline association component for field extraction and data bloodline association processing of the metadata of the standardized cross-domain data set to generate a metadata field parsing result with layered and event identification; a data field parsing result for identifying common association fields and data flow event trigger conditions in different data source metadata to generate a data engineering principle adaptive initial cross-domain association rule; and a test-driven association rule verification component for extracting test samples covering different data types from the standardized cross-domain data set based on the data engineering principle adaptive initial cross-domain association rule to generate a cross-domain data association rule set.

[0041] Preferably, the metadata field parsing is implemented in conjunction with the blood relationship association component: first, the metadata of the standardized cross-domain data set is processed, the core fields are extracted according to the data type (the electronic medical record metadata extracts "patient ID, visit time, department, medical record number", the test report metadata extracts "sample ID, patient ID, test item, test time, report number", and the medical image metadata extracts "image number, patient ID, shooting part, shooting time, device model"), the data type (such as "patient ID" is a string type, and "test time" is a timestamp type) and constraint rules (such as "patient ID" is an 18-digit number + letter combination, and "test item" needs to match a preset test item dictionary) of each field are marked, and a type-specific metadata field list is obtained; then, the type-specific metadata field list is processed, the field source and flow path (such as the "sample ID" of the test report is derived from the "visit application number" of the electronic medical record, and the "patient ID" of the medical image is directly synchronized from the "patient ID" of the electronic medical record) are traced back through a data blood relationship tracking tool (such as Apache Atlas, an open source data governance tool), the "source-target" mapping relationship (such as "electronic medical record. Visit application number → test report. Sample ID" and "electronic medical record. Patient ID → medical image. Patient ID") between fields is recorded, and a metadata blood relationship table is generated; then, the metadata blood relationship table and the data lake storage layering strategy are processed, a layering identifier (such as the electronic medical record field is marked as "hot data layer-visit core data", the test report field is marked as "warm data layer-diagnosis auxiliary data", and the medical image field is marked as "cold data layer-image diagnosis data") is added for each field; at the same time, in combination with the smart medical business events (such as "outpatient visit event", "test sampling event", and "image examination event"), an event identifier (such as "patient ID + visit time" is marked as "outpatient visit event association field", and "sample ID + test time" is marked as "test sampling event association field") is added for the associated fields; finally, the type-specific metadata field list, the metadata blood relationship table, the layering identifier, and the event identifier are integrated to generate a metadata field parsing result with layering and event identifiers. This technology is different from the traditional parsing method of only extracting field names. By adding blood relationship association, layering, and event identifiers, the source and business scenario attributes of the fields can be clearly defined, providing "field meaning + business background" dual basis for subsequent association rule construction, and avoiding the misassociation problem of "same field name but different meaning" in traditional parsing.

[0042] Preferably, the specific implementation process of the engineering principle adaptation type cross-domain association rule generation component is as follows: first, the metadata field parsing result with hierarchical and event identification is taken as the processing object, the coincident fields in different data source metadata are screened through the common field identifier (such as “patient ID” is contained in electronic medical record, test report and medical image metadata, and “shooting / test time” is contained in test report and medical image metadata), and is marked as a candidate association field set; then, the candidate association field set and the event identification are taken as the processing object, the binding relationship between the field and the business event is analyzed: “patient ID” can be associated across all events (outpatient visit, test and image), “visit time” and “test time” need to meet “test time is within 1-72 hours after visit time” (consistent with the logic of “first visit and then test” in the medical scene), “visit department” and “shooting site” need to meet “chest / abdominal image for internal medicine visit, and skeletal image for orthopedics visit” (consistent with the logic of department diagnosis and treatment range), and the logical relationship is extracted as the event triggered association condition; then, the candidate association field set and the event triggered association condition are taken as the processing object, the data engineering core principles are adapted: first, the “consistency principle” requires uniformity of the association field format (such as converting “patient ID” of different data sources into “number + letter” without space format); second, the “timeliness principle” sets the association execution time limit (such as completing cross-data source association within 30 minutes after event triggering to avoid data inconsistency caused by delay); third, the “redundancy exclusion principle” eliminates repeated association logic (such as when “patient ID + visit time” can uniquely associate, “medical record number + report number” is no longer added for redundant association), and the engineering principle constraint condition is generated; finally, the candidate association field set, the event triggered association condition and the engineering principle constraint condition are integrated to form a structured rule of “association field + trigger condition + constraint rule” (such as “rule 1: electronic medical record and test report are associated through “patient ID”, the trigger condition is “test time is within 1-72 hours after visit time”, and the constraint condition is “patient ID format is unified to 18-bit spaceless string”), and the data engineering principle adaptation type initial cross-domain association rule is obtained. This technology is different from the traditional rule generation method based on field name matching only, and by combining business event logic and engineering principles, the rule not only meets the actual business scene demand, but also has engineering landing feasibility, solving the problems of “poor business adaptation” or “engineering execution difficulty” of traditional rules.

[0043] Preferably, in the specific technical implementation of the test-driven association rule verification component: first, the data engineering principle adaptive initial cross-domain association rule and the standardized cross-domain data set are taken as the common processing objects, the sampling strategy is designed according to the data source type (electronic medical record, test report, medical image) and business event (outpatient visit, test, image examination) covered by the rule, and the sample coverage of "field format abnormality" (such as "patient ID" containing spaces), "time range boundary" (such as test time being 1 hour / 72 hours after visit), "department-part matching" (such as internal medicine visit corresponding to skeletal image, i.e. abnormal scene) and other scenes is ensured, the samples are extracted from the standardized cross-domain data set, and a multi-scene test sample set (containing 1000 samples, of which 700 are normal scenes, 200 are boundary scenes, and 100 are abnormal scenes) is generated; then, taking the multi-scene test sample set and the initial cross-domain association rule as the processing objects, an automated test framework is built, and rule verification is performed piece by piece: the metadata fields in the sample are substituted into the rule, it is checked whether the association fields are matched, the trigger conditions are met, and the constraint conditions are compliant, the "matching success rate" (such as rule 1 matching successfully in 680 normal scenes, with a success rate of 97.1%) and "abnormal scene recognition rate" (such as rule 1 identifying 95 abnormal samples of "patient ID containing spaces" with a recognition rate of 95%) of each rule are recorded, and a rule test result report is generated; then, taking the rule test result report as the processing object, setting the verification passing threshold (such as matching success rate ≥ 95%, abnormal scene recognition rate ≥ 90%), screening the rules that do not pass (such as "department-part matching" rule matching success rate is only 80%), analyzing the failure reasons (such as not covering the special scene of "internal medicine emergency may take head image"), modifying the rule content (such as supplementing the exception clause of "when internal medicine emergency, the shooting part can include the head"), and obtaining the modified cross-domain association rule; finally, taking the modified cross-domain association rule and the supplementary test sample (100 new samples for the modified scene) as the processing objects, performing secondary verification to ensure that the modified rule meets the threshold requirements, integrating all the verified rules, and generating a cross-domain data association rule set (containing 5 core rules, each rule is labeled with applicable scene, verification success rate and exception clause). This technology is different from the traditional rule application method without testing directly. Through multi-scene test samples and automated verification, business vulnerabilities and engineering defects in the rule can be found in advance, the accuracy and reliability of the association rule can be greatly improved, and cross-domain data association errors caused by direct use of the rule can be avoided.

[0044] Optionally, the micro-service data fusion processing submodule specifically comprises the following three components: a micro-service instance scheduling component for scheduling corresponding data matching micro-service instances through a container orchestration system based on the data volume of the standardized cross-domain data set to generate micro-service scheduling configuration information; a cross-domain data matching component for performing data matching processing on the standardized cross-domain data set based on the micro-service scheduling configuration information to find data sets in different data sources that meet the association rules to generate a matching data set pair; and a data fusion splicing component for performing fusion splicing processing on the matching data set through a data splicing micro-service instance to generate a preliminary fusion data set.

[0045] Preferably, the microservice instance scheduling component is implemented as follows: first, the standardized cross-domain data set is taken as the processing object, the magnitude of each source data is counted according to the data type (such as 500,000 pieces of store POS sales data, 300,000 pieces of online e-commerce order data, and 200,000 pieces of member APP behavior data), and the data density (such as 20 fields per piece of sales data and 15 fields per piece of order data) is calculated, the matching processing time of a single data is calculated (based on historical data, the average matching time of a single data is 0.5 ms), and the type-specific data processing amount and time consumption evaluation result is obtained; then, the type-specific data processing amount and time consumption evaluation result is taken as the processing object, the preset service instance performance benchmark (a single data matching microservice instance can process 2000 pieces of data per second, occupying 0.2 CPU cores and 256 MB of memory) is combined, the required instance number is preliminarily calculated (such as a total data amount of 1,000,000 pieces, the required instance number = 1,000,000 pieces x 0.5 ms / piece ÷ 1,000 ms / s ÷ 2,000 pieces / s / instance ≈ 0.25, rounded up to 1), and the initial instance number calculation result is generated; then, the initial instance number calculation result and the cluster resource state of the container orchestration system (Kubernetes, referred to as K8s) (such as 1 CPU core and 1 GB of memory remaining in the current cluster) are taken as the processing object, and resource adaptation adjustment is performed: if the remaining resources are insufficient to support the initial instance number (such as 3 instances are required initially, requiring 0.6 CPU cores and 768 MB of memory, and the cluster remaining resources are sufficient), the initial instance number is directly used; if the remaining resources are insufficient (such as 5 instances are required initially, requiring 1 CPU core and 1.25 GB of memory, and the cluster remaining memory is only 1 GB), the instance number is dynamically reduced to 4, and the single instance processing timeout time is extended (from the default 30 seconds to 45 seconds), and the resource-adapted instance configuration is generated; finally, the resource-adapted instance configuration is taken as the processing object, the service orchestration rules of the instances are defined (such as the matching instances of the store POS data and the online order data are deployed in the same availability zone to reduce network latency, and the member behavior data matching instances are deployed separately to avoid resource occupation with sales / order data), and the K8s API is used to create instance deployment tasks, record the IP address, port number, and allocated data processing range of the instances, generate the microservice scheduling configuration information (including instance ID, processing data type, resource quota, deployment location, and processing range). This technology is different from the traditional fixed instance number scheduling method. By combining data amount to dynamically calculate instance demand and cluster resource adaptation, it not only avoids resource waste caused by excessive instances, but also prevents processing delay caused by insufficient instances, solving the problem of "resource and load imbalance" in traditional scheduling.

[0046] Preferably, the specific implementation process of the cross-domain data matching component is as follows: first, the micro-service scheduling configuration information and the standardized cross-domain data set are taken as common processing objects, the standardized cross-domain data set is split into multiple data shards according to the "instance-data range" mapping relationship in the configuration information (for example, the first instance processes the first 200,000 pieces of store POS sales data + the first 100,000 pieces of online order data, and the second instance processes the last 300,000 pieces of store POS sales data + the last 200,000 pieces of online order data + the first 100,000 pieces of member APP behavior data), each shard carries a corresponding cross-domain data association rule set (for example, "member ID consistency + transaction time difference ≤ 5 minutes" is the matching rule for sales data and order data, and "member ID consistency + behavior time within 1 hour before order creation" is the matching rule for order data and member behavior data), and a split-instance data shard and rule binding result is obtained; then, the split-instance data shard and rule binding result is taken as a processing object, and parallel matching threads are started by each micro-service instance: first, a hash index is established for the data in the shard according to the association key field (for example, "member ID") (data with the same member ID is classified to reduce the number of traversals), and then multi-dimensional matching is performed according to the time range, field consistency and other conditions in the rule (for example, the store sales data (transaction time 14:30) of the member ID "VIP001" is matched with the online order data of the same member ID, with a transaction time of 14:25-14:35, and the APP browsing behavior data of the member from 13:30 to 14:30), and a single-instance matching candidate result is obtained; then, the single-instance matching candidate result is taken as a processing object, and matching confidence verification is performed: the matching result that completely meets all rule conditions (such as consistent member ID and time range) is marked as "high confidence (≥ 90%)"; the result that partially meets the rule (such as consistent member ID, but time difference is 6 minutes, exceeding the rule threshold of 1 minute) is marked as "medium confidence (60%-89%)"; the result that only the key field is consistent and other conditions do not meet the rule is marked as "low confidence (<60%)", and a single-instance matching result with confidence is generated; finally, the matching results with confidence of all instances are taken as processing objects, the "high confidence" results are filtered through a result aggregator, the "low confidence" results are removed, the "medium confidence" results are supplemented with secondary verification of business logic (such as judging whether it is an online and offline linked activity order of the same store), and finally a matching data set pair containing "matching data pair ID, source data identifier, matching rule, and confidence" is formed. This technology is different from the traditional single-thread traversal matching method, and through hash index optimization and multi-dimensional confidence verification, it not only improves the matching efficiency, but also reduces the false matching rate of "consistent key field but unrelated business", and solves the problem of "difficulty in balancing efficiency and accuracy" in traditional matching.

[0047] Preferably, in the specific technical implementation of the data fusion splicing component: first, take the matching data set pair and the micro-service scheduling configuration information as the common processing object, based on the service dependency relationship in the configuration information (the data splicing micro-service instance needs to be deployed in the same node as the corresponding data matching micro-service instance to reduce data transmission overhead), call the data splicing micro-service instance in the K8s cluster (according to the matching type, divided into "sales-order splicing micro-service" and "order-member behavior splicing micro-service", respectively adapted to the field structure of different source data), and distribute the matching data set pair to the corresponding splicing instance according to the type to obtain the splicing instance-matching data pair distribution result; then take the splicing instance-matching data pair distribution result as the processing object, and each splicing instance first performs field conflict detection and processing: for the case that the field names are the same but the values are different in the same matching data pair (such as "member points" in the member APP behavior data is 100 points, and "member points" in the online order data is 95 points), according to the conflict handling policy in the cross-domain data association rule set (such as "take the maximum value of the accumulation type field", "take the earliest value of the time type field", and "merge and remove duplicates of the description type field"), perform correction to obtain the data pair after field conflict handling; then take the data pair after field conflict handling as the processing object, and perform splicing according to the business field priority: take the core business fields (such as "member ID, transaction amount, transaction time") as the basic fields of the fusion data and place them in the first segment of the data structure; take the extended fields (such as "store number, APP browsing page, payment method") as supplementary fields and place them after the basic fields; at the same time, add "data source identifier" (such as "transaction amount (source: store POS)" and "browsing page (source: member APP)") to each field to record the original source of the field, and generate structured splicing data; finally, take the structured splicing data of all splicing instances as the processing object, sort them according to the combined key field of "member ID+transaction time" through the data integrator (to ensure that the fusion data of the same member in the same time period is continuous), supplement the fusion metadata (such as fusion timestamp, number of data sources participating in fusion, and splicing micro-service instance ID), remove duplicate fusion records (such as duplicate data generated by splicing the same matching data pair multiple times), and integrate to form the preliminary fusion data set containing complete business fields, source identifiers, and fusion metadata. This technology is different from the traditional simple field superposition fusion method, and through field conflict policy processing and source identifier recording, it not only ensures the consistency of the fusion data, but also realizes data traceability, solving the problems of "field conflict cannot be defined" and "data source cannot be traced" in traditional fusion.

[0048] Optionally, the consistency check sub-module specifically comprises the following three components: a consistency check dimension definition component for analyzing the structural features of the preliminary fused data set to generate a consistency check dimension table; a fused data check execution component for performing record-by-record check processing on the preliminary fused data set based on the consistency check dimension table to generate a to-be-corrected fused data set; and a check problem correction component for processing the check items of the to-be-corrected fused data set using a preset correction strategy to generate the cross-domain fused data set.

[0049] Preferably, the consistency check dimension definition component is specifically implemented as follows: first, taking the preliminary fused data set as the processing object, extracting the basic attributes of all fields (e.g., "order number" is a string type with a fixed length of 12 digits, "purchase quantity" is an integer type with a value ≥ 1, and "warehouse-in time" is a timestamp type and needs to be later than "purchase time"), and marking whether the field is a "core associated field" (e.g., "order number" and "supplier ID" are core associated fields and are used for cross-data-source matching) to obtain a list of basic attributes of the fused data fields; then, taking the list of basic attributes of the fused data fields as the processing object, combining the wisdom supply chain business logic (e.g., "purchase quantity should be ≥ warehouse-in quantity, and the allowed reasonable loss rate ≤ 5%" "when the logistics number is not empty, the warehouse-in time needs to be later than the logistics receipt time" "the inventory stock change amount should be equal to the warehouse-in quantity minus the warehouse-out quantity"), and combing the business association constraints between fields to generate a list of business logic association constraints; then, taking the list of basic attributes of the fused data fields and the list of business logic association constraints as the processing object, defining two types of consistency check dimensions: one is "basic format dimension" (checking whether the field data type, length, and value range are compliant), and the other is "business logic dimension" (checking whether the association constraints between fields are satisfied), and specifying the check rules for each dimension (e.g., the "warehouse-in time" rule in the basic format dimension is "format is YYYY-MM-DD HH:MM:SS and later than the purchase time", and the "purchase and warehouse-in quantity" rule in the business logic dimension is "warehouse-in quantity ≤ purchase quantity × (1-0.05)"), to obtain a preliminary version of the consistency check dimension; finally, taking the preliminary version of the consistency check dimension as the processing object, combining the common problems of historical fused data (e.g., "logistics number is empty but warehouse-in time has been filled" "inventory stock is negative"), supplementing the missing dimensions (e.g., "non-empty field dimension" "inventory value non-negative dimension"), and adjusting the rule details (e.g., the upper limit of the loss rate is refined from 5% to "5% for ordinary goods and 8% for fragile goods"), to finally generate the consistency check dimension table (containing dimension name, check field, check rule, and applicable scenario). This technology is different from the traditional single check dimension design that only focuses on field format. By combining business logic to define multi-dimensional check rules, it can cover both "format error" and "logical contradiction" problems, avoiding the missed detection of "format compliant but business logic invalid" in traditional checks.

[0050] Preferably, the specific implementation process of the fusion data verification execution component is as follows: first, taking the consistency verification dimension table and the preliminary fusion data set as common processing objects, the preliminary fusion data set is divided into pieces according to the "core correlation field + data generation time" (such as splitting according to the "first 6 digits of the order number (representing the supplier) + month", each piece contains the fusion data of the same supplier in the same month), to ensure that the data in each piece has strong business correlation, facilitating centralized verification, and obtaining a fusion data verification piece set; then taking the fusion data verification piece set and the consistency verification dimension table as processing objects, assigning an independent verification thread to each piece, and the thread performs verification according to the order of the verification dimension table: first, performing basic format dimension verification (such as checking whether the "order number" length is 12 digits and whether the "purchase quantity" is a positive integer), marking the fields with format violations and reasons (such as "order number: length 11 digits, not meeting the 12-digit requirement"); then performing business logic dimension verification (such as checking whether the "warehouse-in quantity" is ≤ "purchase quantity" x 95% and whether the "warehouse-in time" is later than the "purchase time"), marking the field combinations with logical contradictions and reasons (such as "purchase quantity 100, warehouse-in quantity 102, exceeding the allowed loss ratio"), and generating single-piece verification problem marking results (including record ID, problem dimension, problem field, and problem description); then taking the single-piece verification problem marking results as processing objects, and performing problem aggregation on each fusion data record: if a record has multiple problems (such as "order number format error + warehouse-in quantity exceeds loss"), then integrate all problem descriptions and mark the record as "to-be-corrected record"; if the record has no problems, then mark it as "compliant record", and obtain the record verification state table within the piece; finally, taking the record verification state table within the piece of all pieces as processing objects, filtering out all "to-be-corrected records" through the result aggregator, associating their original fusion data content (such as "order number: PO20241000001, purchase quantity: 100, warehouse-in quantity: 102, supplier ID: SC001") and complete problem list, sorting according to "problem dimension priority" (business logic problem priority higher than basic format problem), and generating a to-be-corrected fusion data set (including complete data of to-be-corrected records, problem list, and priority). This technology is different from the traditional single-thread record-by-record verification method, and through parallel verification and problem aggregation and sorting of pieces, it not only improves the verification efficiency of large-scale fusion data, but also facilitates subsequent priority processing of key business logic problems through priority sorting, solving the problems of "low efficiency and disordered problems" in traditional verification.

[0051] Preferably, in the specific technical implementation of the verification question correction component: first, take the fusion data set to be corrected as the processing object, divide the questions into "high-priority business logic questions" (such as "warehouse quantity > purchase quantity" and "inventory balance is negative"), "medium-priority mandatory field questions" (such as "core associated field is empty"), and "low-priority format detail questions" (such as "date format is YYYY / MM / DD instead of YYYY-MM-DD") according to the "question dimension priority", and match a preset correction strategy (such as "business rule oriented correction" for high-priority questions, "data trace completion" for medium-priority questions, and "automatic format conversion" for low-priority questions) for each type of question to obtain the question classification and strategy matching result; then take the question classification and strategy matching result as the processing object, and perform targeted correction: for high-priority business logic questions (such as "warehouse quantity 102 > purchase quantity 100, loss rate 2%"), if the warehouse quantity is within the range of "purchase quantity x (1 + 0.02)" (may be due to input error), then correct it to 95 according to the upper limit of "purchase quantity x (1 - 0.05)" (or keep the original value and mark "excessive loss to be manually confirmed"); for medium-priority mandatory field questions (such as "logistics order number is empty"), query the original logistics data of the corresponding purchase order through a data trace tool (such as obtaining the logistics order number from the logistics system interface) to complete the missing field; for low-priority format detail questions (such as "date format YYYY / MM / DD"), automatically convert it to the standard format "YYYY-MM-DD" to generate the preliminary corrected fusion data set; then take the preliminary corrected fusion data set and the consistency verification dimension table as the processing object, and perform full-dimension verification again on the corrected records (to ensure that no new problems are generated after correction): if the corrected records pass all verifications, mark them as "correction completed compliant records"; if there are still problems (such as "logistics order number completion failure"), mark them as "records to be manually intervened" and generate a problem explanation (such as "the logistics order number of this order cannot be obtained from the logistics system, manual checking is required"), to obtain the corrected verification result table; finally, take the corrected verification result table as the processing object, integrate the "correction completed compliant records" with the "compliant records" in the previous verification, remove the invalid "records to be manually intervened" (or store them separately for later processing), sort them according to "order number + warehousing time", supplement the correction records (such as correction field, before and after correction value, correction strategy, correction time), and generate the cross-domain fusion data set. This technology is different from the traditional manual correction method, which greatly reduces the cost of manual intervention through the automatic correction process of "question classification + preset strategy + secondary verification", and ensures the correction quality through secondary verification, solving the problems of "low efficiency and new problems after correction" in traditional correction.

[0052] Optionally, the structured storage module is configured to perform structured storage and analysis modeling processing on the cross-domain fusion data set to generate a cross-domain data fusion analysis result, and specifically includes the following three sub-modules: a fusion data structured import sub-module configured to perform structured processing on the cross-domain fusion data set and import the data into a data warehouse to generate structured storage fusion data; a fusion data modeling analysis sub-module configured to call a pre-trained machine learning model based on the structured storage fusion data to perform analysis processing to generate a preliminary data fusion analysis result; and an analysis result integration processing sub-module configured to perform integration and optimization processing on the preliminary data fusion analysis result, remove redundant information, supplement associated descriptions, and generate a cross-domain data fusion analysis result.

[0053] Optionally, the fusion data structured import sub-module is configured to perform structured processing on the cross-domain fusion data set and import the data into a data warehouse to generate structured storage fusion data, and specifically includes the following three components: a data warehouse table structure adaptation component configured to compare the field structure of the cross-domain fusion data set with the table structure of the target data warehouse to generate a field mapping relationship table; a structured data conversion component configured to perform structured conversion processing on the cross-domain fusion data set based on the field mapping relationship table to generate structured import data; and a data warehouse write component configured to call a write interface of the data warehouse, write the structured import data into a corresponding table according to a pre-set partitioning strategy (such as partitioning according to the collection date) to generate structured storage fusion data.

[0054] Preferably, the data warehouse table structure adaptation component is specifically implemented as follows: firstly, taking the cross-domain fused data set as the processing object, the structural characteristics of all fields are extracted, including the field name (such as "photovoltaic output value", "fan rotating speed", "energy storage SOC value"), data type (such as floating point type, integer type, percentage type), field length (such as "photovoltaic output value" with 2 decimal places and 6 bytes in length), and business meaning (such as "energy storage SOC value" representing the state of charge of the energy storage system), to obtain a list of fused data field structures; secondly, taking the table structure of the target data warehouse as the processing object, the field definitions of each table (such as "new energy comprehensive monitoring table" containing "timestamp, station ID, energy type, real-time output, and device status code", and "device operation and maintenance analysis table" containing "device ID, collection time, running parameter, and fault warning value") are parsed, and field constraints (such as "real-time output" being required to be ≥ 0 and "device status code" being a 3-digit integer) are recorded to generate a list of data warehouse table structures; thirdly, taking the list of fused data field structures and the list of data warehouse table structures as common processing objects, field semantic comparison is performed: for fields with different names but consistent business meanings (such as "photovoltaic output value" of the fused data and "real-time output (energy type = photovoltaic)" of the warehouse table), they are marked as "semantically matched fields"; for fields with the same name but different meanings (such as "device status" of the fused data indicating the running mode and "device status" of the warehouse table indicating the fault level), they are marked as "homonymic heteronymic fields"; for fields existing only on one side, they are marked as "newly added / missing fields", to obtain the field semantic comparison result; finally, taking the field semantic comparison result as the processing object, direct mapping is established for the semantically matched fields (such as "photovoltaic output value → real-time output, with the energy type being filled with 'photovoltaic' by default"); alias mapping is added for the homonymic heteronymic fields (such as "fused device status → running mode, and warehouse device status → fault level"); and it is evaluated whether the newly added fields need to extend the warehouse table structure (such as "energy storage charging and discharging efficiency" being a key indicator, and the "new energy comprehensive monitoring table" is extended to add the field), to integrate and form a field mapping relationship table containing "fused field name, warehouse field name, mapping type, and conversion specification". This technology is different from the traditional simple comparison based on field names, and through the introduction of business semantic analysis, it can accurately identify homonymic heteronymic or heteronymic homonymic fields, avoiding the problem of data warehouse storage disorder caused by field mis-mapping.

[0055] Preferably, the specific implementation process of the structured data conversion component is as follows: first, taking the field mapping relationship table and the cross-domain fusion data set as common processing objects, according to the corresponding relationship of "fusion field name-warehouse field name" in the mapping relationship table, the cross-domain fusion data set is split into multiple sub-data sets (such as a sub-data set matching the "new energy comprehensive monitoring table" and a sub-data set matching the "device operation and maintenance analysis table"), each sub-data set contains only the required fields of the corresponding table, and a table field screening result is obtained; then taking the table field screening result and the field mapping relationship table as processing objects, performing data type conversion: converting numerical value type fields (such as the "fan speed (string type)" of the fusion data) into integer type according to the requirements of the warehouse table (retaining the original numerical value and eliminating non-numeric characters); converting time fields (such as the "collection time (millisecond timestamp)" of the fusion data) into "YYYY-MM-DD HH:MM:SS" format strings of the warehouse table; converting enumeration type fields (such as the "device state ('running''stop')" of the fusion data) into code values of the warehouse table (such as "1" representing running and "0" representing stop), generating a type-converted sub-data set; then taking the type-converted sub-data set as a processing object, performing field format standardization: uniformly retaining the precision of numerical value type fields (such as "photovoltaic output value" retaining 2 decimal places and "energy storage SOC value" retaining 1 decimal place); removing special characters from text type fields (such as uniformly replacing spaces and underscores in "device ID" with no signs); supplementing associated fields with association identifiers (such as uniformly prefixing "site ID" with "P-" representing a photovoltaic site and "W-" representing a wind power site), obtaining a format-standardized sub-data set; finally, taking all format-standardized sub-data sets as processing objects, adjusting the field order according to the table structure requirements of the target data warehouse (such as "new energy comprehensive monitoring table" sorted by "timestamp-> site ID-> energy type-> real-time output"), supplementing the default fields of the warehouse table that are required to be filled but missing in the fusion data (such as "data source" filled with "cross-domain fusion" by default), and integrating to form structured data to be imported (each sub-data set corresponds to a table in the warehouse, and the field structure, type, and format are completely adapted). This technology is different from the traditional fixed script conversion method, and through the hierarchical processing of table screening, type conversion, and format standardization, it can adapt to the individual structure requirements of different data warehouse tables while ensuring the consistency of the converted data, solving the problem of "one table one script, poor scalability" in traditional conversion.

[0056] Preferably, in the specific technical implementation of the data warehouse writing component: first, the partition strategy of the structured to-be-imported data and the target data warehouse (such as the "new energy comprehensive monitoring table" is partitioned according to the "collection date (YYYY-MM-DD)", and the "device operation and maintenance analysis table" is partitioned according to the "device type + quarter") is taken as a common processing object, the partition key field (such as "collection time" and "device type") in the structured to-be-imported data is analyzed, the data is split according to the partition rule (such as writing the photovoltaic data of October 15 into the "new energy comprehensive monitoring table_2024-10-15" partition, and writing the three-quarter data of the fan device into the "device operation and maintenance analysis table_ fan_Q3" partition), and a partition data shard set is obtained; then, the partition data shard set is taken as a processing object, and the writing mode of each partition is judged: the full-amount writing mode (directly writing all data) is adopted for a new partition (such as the "2024-10-15" partition first written on the same day); the incremental writing mode (first comparing the data primary key, and only writing the record that does not exist or is updated) is adopted for an existing partition (such as supplementally writing the missing data of October 15), and a partition writing mode configuration is generated; then, the partition data shard set and the partition writing mode configuration are taken as processing objects, a special writing interface (such as the HDFS (Hadoop Distributed File System) API of the Hive data warehouse and the JDBC (Java Database Connectivity) interface of the MySQL data warehouse) of the target data warehouse is called, and writing is performed according to the configuration: the batch submission is enabled when full-amount writing (every 1000 data is submitted once, reducing the number of interface calls); the primary key duplication checking (such as "timestamp + station ID" as the joint primary key) is performed first, and then the differential data is written when incremental writing, and the writing state ("writing success", "primary key conflict", "format error") of each data is recorded, and a writing state record table is obtained; finally, the writing state record table and the partition data shard set are taken as processing objects, the partition information of the "writing success" data is associated, a writing success data list is generated; the "primary key conflict" data is marked as "duplicate data" (not written repeatedly); the "format error" data (such as abnormal values missing in conversion) is stored in the "error data buffer" (re-written after manual correction), the writing success data list is integrated, it is confirmed that the corresponding table and partition in the target data warehouse have stored complete and consistent data, and the fusion data of the structured storage is generated. This technology is different from the traditional full-amount covering writing mode, and through the combination of dynamic partition splitting and incremental writing, the resource consumption of large-scale data writing is reduced, the storage space occupied by duplicate data is avoided, and the problems of "low efficiency and data redundancy" in the traditional writing are solved.

[0057] Optionally, the fusion data modeling analysis submodule specifically comprises the following three components: a model selection and adaptation component: configured to select an adapted pre-trained machine learning model from a pre-set model library according to an analysis requirement description of the structured stored fusion data, and configure model input parameters to generate model configuration information; a model data input component: configured to perform feature extraction and format conversion processing on the structured stored fusion data based on the model configuration information to generate model input data; and a model analysis execution component: configured to call the pre-trained machine learning model to perform analysis processing based on the model configuration information and the model input data, and output a model analysis result to generate a preliminary data fusion analysis result.

[0058] Preferably, the model selection adaptation component is implemented as follows: first, the structured analysis requirement description of the fusion data is taken as the processing object, the core task type (such as "fault probability prediction" belongs to a combination of classification and regression tasks) and the data dimension (time series data containing 3 types of sensor 10-minute level sampling values) in the requirement are parsed, and the output requirement (probability value 0-1 with feature importance ranking) is obtained to obtain a requirement task feature list; then, the requirement task feature list is taken as the processing object, a preset model library (containing a long short-term memory network LSTM (a time series processing neural network), a gradient boosting tree GBDT (an integrated learning model), and a convolutional neural network CNN (a feature extraction neural network)) is called, and the adaptation scenarios of each model (LSTM adapts to long time series dependent data, GBDT adapts to nonlinear feature interaction data, and CNN adapts to data with significant local features) are labeled to generate a model library adaptation scenario table; then, the requirement task feature list and the model library adaptation scenario table are taken as the processing objects, and multi-dimensional matching is performed: because the analysis requirement involves 24-hour time series prediction (long time series dependence), and the vibration data contains periodic local fluctuations (local features), a combination model of LSTM and CNN (considering time series and local features) is preliminarily screened; at the same time, the pre-training effect of the model on the manufacturing scene is verified (such as a model containing 100,000+ motor fault samples in the pre-training data is preferred), and a candidate model set is obtained; finally, the candidate model set is taken as the processing object, and the model input parameters are configured: the input feature dimension (vibration amplitude, temperature value, and energy consumption value, a total of 3 dimensions), the time series window size (taking the past 24-hour data, that is, 144 10-minute sampling points), the output layer activation function (sigmoid function, output 0-1 probability), and the iteration number (default 100 times, which can be dynamically adjusted according to the loss value) are determined, and the model configuration information (containing the model name, structure parameter, input dimension, and pre-training weight path) is integrated. This technology is different from the traditional model selection method according to the task type, and through multi-dimensional adaptation combining the time series characteristics of the data and the pre-training effect of the scene, the prediction deviation caused by the mismatch between the model and the data characteristics is avoided, and the pertinence of model selection is improved.

[0059] Preferably, the specific implementation process of the model data input component is as follows: first, take the model configuration information and the structured stored fusion data as common processing objects, extract the target fields (such as "vibration amplitude (mm / s)", "temperature (°C)", and "energy consumption (kW·h)") from the structured stored fusion data according to the "input feature dimension" in the configuration information, filter invalid records (such as null records caused by sensor offline, filled with the average of the previous and next 3 data), and obtain the original feature data set; then, take the original feature data set as the processing object, perform feature enhancement processing: calculate the derived indicators (such as the 5-minute sliding average of the vibration amplitude, the temperature rise rate (°C / h), and the standard deviation of the energy consumption) for the time series features, enrich the feature dimension (from 3 dimensions to 9 dimensions), and perform one-hot encoding (such as model A coded as [1, 0, 0] and model B coded as [0, 1, 0]) for the category type implicit features (such as device model, extracted from "device ID"), and generate the enhanced feature data set; then, take the enhanced feature data set and the model configuration information as the processing objects, perform format conversion: according to the "time series window size" (144 sampling points) in the configuration information, divide the enhanced feature data set into fixed-length sample sequences (each sample contains a 144x9 feature matrix, with rows representing sampling time points and columns representing feature dimensions); convert the sample sequence into a model-compatible tensor format (such as the [sample number, time step, feature number] three-dimensional tensor required by LSTM), and normalize the feature values (scaled to the 0-1 interval according to the mean and standard deviation of the training set, to avoid the influence of different orders of magnitude features on the model weight), to obtain the tensor format feature data; finally, take the tensor format feature data as the processing object, divide the input data subsets: according to the 7:2:1 ratio, split the training and validation set (used for model fine-tuning), the test set (used for effect evaluation), and the prediction set (the latest 24-hour data to be predicted), and add data identifiers (such as "training_deviceID_M01" and "prediction_time_20241015") to each subset, to generate model input data (including tensor data and identifier information of each subset). This technology is different from the traditional direct use of original features. Through feature enhancement and dynamic format adaptation, it not only supplements the implicit features required by the model, but also ensures that the input format is completely matched with the model structure, solving the problem of "insufficient original feature dimension or format mismatch leading to model performance decline".

[0060] Preferably, in the specific technical implementation of the model analysis execution component: first, take the model configuration information and the model input data as the common processing objects, load the pre-trained machine learning model (such as the pre-trained weight file of the LSTM-CNN combined model) in the configuration information, pass the training and validation set in the model input data into the model, perform fine-tuning training (use small batch gradient descent, batch size is set to 32, learning rate is initially 0.001, and is attenuated by 10% every 20 rounds), adjust the model parameters through the loss value (cross-entropy loss) of the validation set (stop fine-tuning when the loss value does not decrease for 5 consecutive rounds), and obtain the fine-tuned model; then take the fine-tuned model and the test set in the model input data as the processing objects, perform model effect evaluation: calculate the prediction accuracy (correct proportion of fault samples), precision (proportion of actual faults in samples predicted as faults), and recall (proportion of predicted faults in actual fault samples) of the test set, and if the evaluation indicators are all higher than the preset threshold (such as accuracy ≥ 85%, recall ≥ 80%), it is determined that the model is usable; otherwise, return to the model selection and adaptation component to select a model again, and generate a model evaluation result; then take the fine-tuned model that passes the evaluation and the prediction set in the model input data as the processing objects, call the model inference interface to perform analysis and processing: input the prediction set tensor data into the model, output the fault probability value (0-1) and feature importance score (such as the importance proportion of vibration amplitude is 40% and the temperature is 35%) of each prediction sample, and obtain the model original output result; finally, take the model original output result as the processing object, and perform result regularization: mark the samples with a probability value ≥ 0.6 as “high-risk early warning”, mark the samples with a probability value of 0.3-0.6 as “medium-risk attention”, and mark the samples with a probability value < 0.3 as “low-risk normal”; associate the corresponding device ID, prediction time, and feature importance score, and generate a preliminary data fusion analysis result containing early warning levels, triggering features, and confidence level descriptions. This technology is different from the traditional way of directly using a pre-trained model for inference. Through scenario-based fine-tuning and multi-index evaluation, the model is more suitable for the characteristics of the current fusion data. At the same time, through result grading and feature tracing, the explainability of the analysis result is improved, solving the problems of “insufficient generalization of pre-trained models or difficult to understand the results”.

[0061] Optionally, the analysis result integration processing submodule specifically includes the following three components: an analysis result redundancy elimination component: used for identifying and processing redundant information of the preliminary data fusion analysis result, eliminating repeated analysis conclusions and irrelevant intermediate calculation data, to generate a de-redundant analysis result; a result association information supplement component: used for extracting the original data identifier corresponding to the analysis result from the structured stored fusion data, and supplementing it to the de-redundant analysis result, to generate an analysis result with associated identifier; and a result format regularization component: used for performing format regularization processing on the analysis result with associated identifier, to generate a cross-domain data fusion analysis result.

[0062] Preferably, the analysis result redundancy elimination component is specifically implemented as follows: first, taking the preliminary data fusion analysis results as the processing object, the structured information of the results (such as each result containing "analysis type, object identification, conclusion content, generation time, intermediate data list") is parsed, and "analysis type + object identification + generation time" is extracted as the redundancy judgment primary key (such as "congestion prediction + intersection ID = 101 + 2024-10-15 08:00"), and a result primary key and content mapping table is obtained; then taking the result primary key and content mapping table as the processing object, repeated conclusion recognition is performed: for multiple results with the same primary key, the core semantics of the conclusion contents are compared (such as "congestion probability 85%" and "congestion risk is high (85%) " are determined as semantic repetition), and the latest generated one is retained, and the rest are marked as "repeated redundant conclusion"; at the same time, irrelevant intermediate data (such as "feature weight temporary value" and "iteration error record" in the model calculation process, which do not affect the final conclusion and have no reuse value) are identified and marked as "intermediate redundant data", and a redundancy information marked result is generated; then taking the redundancy information marked result and the preliminary data fusion analysis result as the processing object, all complete result entries marked as "repeated redundant conclusion" are eliminated, the fields marked as "intermediate redundant data" in each result are deleted, the core conclusion, key parameters (such as the confidence of the congestion prediction), and analysis basis summary are retained, and a preliminary de-redundancy result is obtained; finally, taking the preliminary de-redundancy result as the processing object, secondary verification is performed through business rules (such as if the "signal state is normal" conclusion appears continuously for more than 3 times, only the first and the latest one are retained, and the intermediate results are marked as "periodic redundancy" and eliminated), and finally a de-redundancy analysis result (each result is unique, has no repeated conclusion, and has no irrelevant intermediate data) is generated. This technology is different from the traditional de-duplication method based on only text complete matching, and through the combination of semantic comparison and business rules, it can accurately identify "different expressions but consistent meanings" redundant conclusions, eliminate valueless intermediate data, and solve the problems of "missing semantic repetition" and "retaining invalid data" in traditional de-redundancy.

[0063] Preferably, the specific implementation process of the result association information supplement component is as follows: first, the de-redundancy analysis result and the structured stored fusion data are taken as common processing objects, the "object identifier" and "analysis type" (such as "object identifier = intersection ID = 101, analysis type = congestion prediction") in the de-redundancy analysis result are parsed, the associated original data range (such as the camera traffic flow data, signal timing data, and historical congestion records of the intersection, which are all stored in the "intersection traffic comprehensive table" of the structured fusion data) is determined, and an associated data range list is generated; then, the associated data range list and the structured stored fusion data are taken as processing objects, and the original data identifiers are extracted: for the congestion prediction result, the corresponding "traffic flow data collection timestamp", "signal timing scheme ID", and "historical same period congestion record ID" are extracted; for the device state evaluation result, the "sensor collection ID" and "device maintenance record number" are extracted, to ensure that each identifier can be uniquely traced back to a specific entry in the structured stored fusion data, and an original data identifier set is obtained; then, the original data identifier set and the de-redundancy analysis result are taken as processing objects, and a mapping relationship between the identifiers and the results (such as "congestion prediction result ID = P1001" is associated with "traffic flow data collection timestamp = 2024-10-15 07:30, timing scheme ID = S005") is established, these identifiers are added to the corresponding entries of the de-redundancy analysis result as "data traceability fields", and a result with temporary association identifiers is generated; finally, the result with temporary association identifiers is taken as a processing object, the validity of the identifiers is verified (by querying the structured stored fusion data through the identifiers, it is confirmed that the corresponding original data can be found), invalid identifiers (such as identifiers corresponding to original data that have been excluded due to data cleaning) are marked with "traceability failure" and the reasons (such as "original data is an abnormal value and has been excluded") are supplemented, and the analysis result with association identifiers (including core conclusions, key parameters, valid original data identifiers, and traceability status) is integrated. This technology is different from the traditional fixed supplement of a small number of identifiers, and through dynamic matching of the required original data identifiers according to the analysis type, the pertinence and traceability of the association information are ensured, and the problem of "difficult result traceability due to insufficient or irrelevant association identifiers" is solved.

[0064] Preferably, in the specific technical implementation of the result format regularization component: first, the analysis results with associated identifiers are taken as the processing objects, the application scenario requirements of the results (such as traffic management large-screen display requiring simple and intuitive chart data, decision report generation requiring detailed textual description and data support) are analyzed, two types of basic format templates are defined: “display type template” (containing “object, conclusion, risk level, core data” four fields, and the field values are short sentences or numerical values), “report type template” (containing “analysis object, analysis process summary, conclusion details, associated data identifier, suggestion measures” five fields, and the field values are detailed texts), and a set of scenario-based format templates is obtained; then, the analysis results with associated identifiers and the set of scenario-based format templates are taken as the processing objects, the format templates are allocated according to the emergency level and audience type of the analysis results (such as “high-risk congestion warning” preferentially allocating “display type template” for large-screen real-time pushing, and “monthly traffic situation analysis” allocating “report type template” for written summary), the field contents in the results are filled according to the template requirements (such as the “conclusion details” field integrating core conclusions and key parameters, and the “suggestion measures” field automatically generating basic suggestions according to the conclusions, such as “congestion warning suggestion for temporarily adjusting the timing scheme”), and a preliminary format regularization result is generated; then, the preliminary format regularization result is taken as the processing object, and format uniformity verification is performed: checking whether the field naming of the results of the same type is consistent (such as “object identifier” being unified as “analysis object ID”), whether the numerical value format is standardized (such as the probability value being unified to retain one decimal place, and the time format being unified to “YYYY-MM-DD HH:MM”), and whether the text description conforms to the scene language (such as display type results avoiding professional terms and using “congestion is relatively serious” instead of “congestion probability 85%”), and the inconsistent items are corrected to obtain the result after format verification; finally, the result after format verification is taken as the processing object, the result metadata (such as “generation time, analysis model version, data source quantity”) is added, the analysis types and time stamps are sorted (such as displaying high-risk warnings first, and then displaying regular analysis), and cross-domain data fusion analysis results (containing two types of display type and report type, both of which are uniformly formatted and information complete) that can be directly used for multi-scene output are integrated and formed. This technology is different from the traditional single format regularization method, and through scenario-based templates and dynamic adaptation, the analysis results can directly meet the needs of different application scenarios, and through uniformity verification, the standardization of the results is ensured, solving the problem of “chaotic format requiring secondary processing”.

[0065] Optionally, the fusion analysis result output packaging module specifically comprises the following three sub-modules: an analysis result visualization generation sub-module, configured to perform visualization processing on the cross-domain data fusion analysis result to generate a visualized analysis result; an API interface packaging sub-module, configured to perform API interface packaging processing on the visualized analysis result, to provide an interface for external system calling, so as to generate a callable analysis result API interface; and an interactive result adaptation processing sub-module, configured to adapt the callable analysis result API interface to interactive requirement descriptions of different external systems, so as to generate an interactive cross-domain data fusion analysis output result.

[0066] Optionally, the API interface packaging sub-module specifically comprises the following three components: an API interface specification definition component, configured to define calling parameters of the API interface according to data source hierarchical identification of the visualized analysis result, to generate an API interface specification with version identification; an API interface logic component, configured to generate containerized deployment API interface logic code in a manner of adapting an event-driven architecture based on the API interface specification with version identification; and an API interface publishing component, configured to perform scenario-based publishing processing on the API interface logic code, to generate the callable analysis result API interface.

[0067] Preferably, the API interface specification definition component specifically implements the following: first, taking the hierarchical identification of the data source of the visual analysis result as the processing object, parsing the access characteristics of each layer of data (hot data layer data requires low delay access, response time requirement < 100 ms; warm data layer data allows moderate delay, response time < 500 ms; cold data layer data can be high delay, response time < 2 s) and data permissions (for example, "user credit risk rating" can only be accessed by the internal risk control system, and "asset allocation suggestion" can be opened to the user end APP), obtaining the data hierarchical access characteristics and permission list; then taking the data hierarchical access characteristics and permission list and the visual analysis result type (such as "risk rating", "abnormal detection", and "asset allocation") as the common processing object, defining the core calling parameters of the API interface: mandatory parameters (such as "user ID", "result type", and "data hierarchical identification", which ensure the positioning of unique results), optional parameters (such as "time range" and "return format (JSON / XML)", which meet individual needs), permission parameters (such as "interface calling token Token", which verifies the permission of the calling party), and explicitly defining the type (such as "user ID" is a string type with a length of 18 bits), value range (such as "result type" is only allowed to be "risk_rating", "transaction_detection", and "asset_allocation"), and default value (such as "return format" is JSON by default) of each parameter, generating an initial interface parameter definition table; then taking the initial interface parameter definition table as the processing object, designing version control rules: the version number uses the format of "main version number. Sub version number" (the change of the main version number represents the compatible interruption of the parameter structure, such as the addition or deletion of mandatory parameters; the change of the sub version number represents the compatible upgrade, such as the addition of optional parameters), and marking the version effective time and compatibility strategy (such as V1.0 version can be compatible with V1.1 version calling request, automatically ignoring the added optional parameters), obtaining the parameter definition table with version; finally, taking the parameter definition table with version as the processing object, supplementing the interface interaction specification (such as the request method adopts GET / POST, the response code rule is "200-success", "400-parameter error", "403-insufficient permission", and "500-service exception", and the response data structure includes "status code, result data, hierarchical identification, and generation time"), and integrating to form the API interface specification with version identification (such as "risk_rating_api-V1.0" and "transaction_detection_api-V1.0", each specification includes parameter definition, interaction specification, version description, and permission requirement). This technology is different from the traditional interface specification design that does not associate data hierarchy. By combining data access characteristics to define parameters and permissions, it not only ensures the access efficiency adaptation of different hierarchical data, but also avoids permission boundary calling, solving the problem of "parameter redundancy or permission control missing" in the traditional specification.

[0068] Preferably, the specific implementation process of the API interface logic component is as follows: first, the API interface specification with version identification is taken as the processing object, the parameter definition, interaction specification and data layer identification in the specification are parsed, and the core business logic of the interface is sorted out: receiving the call request→ verifying the parameter legality (such as whether the "user ID" format is correct and whether the "Token" is valid)→ routing to the corresponding data layer according to the "data layer identification" (the hot data layer calls the memory database interface, the warm data layer calls the relational database interface, and the cold data layer calls the object storage interface)→ obtaining the visual analysis result raw data→ converting the data according to the "return format"→ encapsulating the response result (including the status code and result data), and generating the interface business logic flowchart; then, the interface business logic flowchart is taken as the processing object, the business logic is split into independent event nodes (such as "parameter verification event", "data layer routing event", "format conversion event" and "response encapsulation event") according to the requirements of the adaptive event-driven architecture (Event-Driven Architecture, EDA), the event triggering relationship between the nodes is defined (such as triggering the "data layer routing event" after the "parameter verification event" succeeds, and triggering the "response encapsulation event" directly to return error information after the "parameter verification event" fails), and the resource dependency of each event is configured (such as the "data layer routing event" depends on the access SDK of the corresponding data layer), and the event-driven logic splitting result is obtained; then, the event-driven logic splitting result is taken as the processing object, and the API interface logic code is written by using the containerized development framework (such as Spring Boot+Docker): the code contains event handlers (corresponding to the processing logic of each event node, such as ParamValidateHandler, DataRouteHandler), an event bus (responsible for event distribution and node communication), and an exception handler (unified capture of parameter errors, insufficient permissions, data acquisition failures and other exceptions, and return of corresponding response code according to the specification), and version compatibility logic is embedded in the code (such as identifying the version number of the call request, and automatically shielding the optional parameter processing logic added in V1.1 for V1.0 request), and a containerized API interface code package (containing code files, dependency package list pom.xml, and Dockerfile build script) is generated; finally, the containerized API interface code package is taken as the processing object, and syntax checking and logic verification are performed (such as checking whether the parameter verification rules are complete and whether the data layer routing logic covers all layer identifications), and the syntax errors and logic vulnerabilities in the code are corrected (such as supplementing the retry logic for "cold data layer data acquisition timeout"), and the deployable containerized API interface logic code is obtained. This technology is different from the traditional monolithic interface logic development. Through event-driven splitting and containerized packaging, the scalability of the interface logic is improved (new event nodes do not need to reconstruct the whole code), and it is convenient to dynamically expand instances based on the container orchestration system in the future, solving the problems of "high logic coupling degree and difficult deployment and maintenance" in the traditional interface.

[0069] Preferably, in the specific technical implementation of the API interface publishing component: first, taking the deployable containerized API interface logic code and the API interface specification with version identification as the common processing object, combining with the interface use demand of the smart financial risk control scene, dividing the publishing scene type: "internal private publishing" (such as "user credit risk rating API" is published only to the internal risk control system cluster and is not exposed to the outside), "third-party cooperation publishing" (such as "transaction exception detection API" is published to the system of cooperative bank and needs IP whitelist restriction), "user end open publishing" (such as "asset allocation suggestion API" is published to the user APP and needs OAuth2.0 authorization), clearly defining the publishing range (cluster node, IP whitelist), access control policy (Token verification, IP restriction, authorization range) and monitoring index (call volume, response time, error rate) of each scene, and generating a scene-based publishing strategy table; then taking the scene-based publishing strategy table and the containerized API interface logic code as the processing object, performing pre-publishing test based on the container orchestration system Kubernetes (K8s): deploying the code to the test environment K8s cluster, simulating different scene calling requests (such as internal system calling, third-party IP calling, user end authorization calling), verifying the parameter compatibility of the interface (such as whether V1.0 request can respond normally), the validity of permission control (such as whether non-whitelist IP calling returns 403), and whether the response time meets the access characteristic requirements of layered data (such as whether the response time of hot data layer interface is <100ms), recording the test results and correcting the configuration deviation in the publishing strategy (such as supplementing the missing cooperative bank IP in the IP whitelist of "third-party cooperation publishing"), obtaining the verified scene-based publishing strategy; then taking the verified scene-based publishing strategy and the containerized API interface logic code as the processing object, performing K8s cluster publishing: for "internal private publishing", deploying API container instances to internal cluster nodes through K8s Deployment, configuring ClusterIP type Service (only accessible within the cluster); for "third-party cooperation publishing", deploying container instances to edge cluster nodes, configuring NodePort type Service and binding IP whitelist; for "user end open publishing", deploying container instances to cluster nodes accessible from public network, configuring Ingress rules to realize HTTPS encrypted access, and integrating OAuth2.0 Authorization Service; Simultaneously, configure monitoring and alarms for each deployment scenario (e.g., trigger email alarms when the call error rate is >1%, trigger SMS alarms when the response time is >500ms), generate the API interface K8s deployment configuration list (including Deployment, Service, Ingress, and ConfigMap configuration files); Finally, using the API interface K8s deployment configuration list as the processing object, execute the K8s resource creation command (kubectl apply -f deployment configuration list) to deploy the API interface to the corresponding cluster node. After deployment, confirm the container instance running status (Running status) through the K8s Dashboard, and call the interface testing tool Postman to send test requests to verify that the interface can return results normally (response code 200, data format conforms to specifications), and finally generate a callable analysis result API interface (including interface access address, version number, call parameter example, access control requirements, and monitoring and alarm address). This technology differs from traditional unified deployments without scenario-specific approaches. By combining scenario-based deployment strategies with Kubernetes container orchestration, it achieves precise access control for interfaces in different scenarios while ensuring high availability (dynamic scaling based on Kubernetes). This solves the problems of "lax access control and inability to elastically adjust resources" in traditional deployments.

[0070] Figure 2 This is a flowchart illustrating a data weaving method supporting cross-domain data fusion analysis, as described in an embodiment of this application. Figure 2 The process includes: performing distributed asynchronous ingestion processing on the original cross-domain dataset to generate an original data storage pool; performing multi-dimensional data cleaning and standardization processing on the cross-domain data in the original data storage pool to generate a standardized cross-domain dataset; performing cross-domain association and fusion processing on the standardized cross-domain dataset to generate a cross-domain fused dataset; performing structured storage and analytical modeling processing on the cross-domain fused dataset to generate cross-domain data fusion analysis results; and performing visualization output and interface encapsulation processing on the cross-domain data fusion analysis results to generate interactive cross-domain data fusion analysis output results.

[0071] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data weaving apparatus supporting cross-domain data fusion analysis, characterized by, Comprise: Cross-domain data access storage module: for the original cross-domain data set for distributed asynchronous intake processing, to generate the original data storage pool; Multi-dimensional data preprocessing module: for the cross-domain data in the original data storage pool for multi-dimensional data cleaning and standardization processing, to generate the standardized cross-domain data set; Cross-domain data association fusion module: for the standardized cross-domain data set for cross-domain association fusion processing, to generate the cross-domain fusion data set; Structured storage module: for the cross-domain fusion data set for structured storage and analysis modeling processing to generate cross-domain data fusion analysis results; Fusion analysis result output encapsulation module, for the cross-domain data fusion analysis results for visual output and interface encapsulation processing, to generate the interactive cross-domain data fusion analysis output results.

2. The apparatus of claim 1, wherein, Cross-domain data access storage module specifically includes the following three sub-modules: Multi-source data protocol adaptation access sub-module: for the original cross-domain data set for multi-protocol asynchronous access processing, adapting different data source communication protocols to realize real-time / batch asynchronous access, to generate the preliminary access cross-domain data stream; Cross-domain data verification sub-module: for the cross-domain data stream for format legality verification and temporary storage processing, to generate the format compliant temporary cross-domain data set; Cross-domain data hierarchical cloud storage sub-module: for the format compliant temporary cross-domain data set for classification storage scheduling processing, based on data type storage to hierarchical structure directory, to generate the original data storage pool.

3. The apparatus of claim 1, wherein, Multi-dimensional data preprocessing module specifically includes the following three sub-modules: Data anomaly detection processing sub-module: for the cross-domain data in the original data storage pool for anomaly identification and elimination processing, to generate the standardized data after eliminating abnormal; Data format standardization sub-module: for the standardized data after eliminating abnormal for format unification processing, the field structure, data type of different source data unified to the preset standard format, to generate the standardized data of uniform format; Data compression optimization sub-module: for the standardized data of uniform format for compression processing, to generate the standardized cross-domain data set.

4. The apparatus of claim 1, wherein, Cross-domain data association fusion module specifically includes the following three sub-modules: Metadata association rule construction sub-module: for the metadata of the standardized cross-domain data set for parsing and association rule construction processing, determine the association field between different data source data, to generate the cross-domain data association rule set; Micro-service data fusion processing sub-module: for the standardized cross-domain data set based on cross-domain data association rule set, through the container orchestration system scheduling different function microservices for fusion, to generate the preliminary fusion data set; Consistency verification sub-module: for the preliminary fusion data set for consistency verification processing, check the field conflict, logical contradiction of the fused data and modify, to generate the cross-domain fusion data set.

5. The apparatus of claim 4, wherein, Metadata association rule construction sub-module specifically includes the following three components: Metadata field parsing and blood relationship association component: for the metadata of the standardized cross-domain data set for field extraction and data blood relationship association processing, to generate the metadata field parsing results with hierarchical and event identification; Engineering principle adaptive cross-domain association rule generation component: used for identifying common association fields in different data source metadata and data flow event triggering conditions based on metadata field parsing results with hierarchical and event identification, to generate data engineering principle adaptive initial cross-domain association rules; Test-driven association rule verification component: used for extracting test samples covering different data types from standardized cross-domain data sets based on data engineering principle adaptive initial cross-domain association rules, to generate cross-domain data association rule sets.

6. The apparatus of claim 4, wherein, The micro-service data fusion processing sub-module specifically includes the following three components: Micro-service instance scheduling component: used for scheduling corresponding data matching micro-service instances through a container orchestration system based on the data volume of the standardized cross-domain data set, to generate micro-service scheduling configuration information; Cross-domain data matching component: used for data matching processing of the standardized cross-domain data set based on the micro-service scheduling configuration information, to find data sets in different data sources that meet the association rules, to generate a matching data set pair; Data fusion splicing component: used for fusion splicing processing of the matching data set through a data splicing micro-service instance, to generate a preliminary fusion data set.

7. The apparatus of claim 1, wherein, The structured storage module specifically includes the following three sub-modules: Fusion data structured import sub-module: used for structured processing of the cross-domain fusion data set and importing it into a data warehouse, to generate structured storage fusion data; Fusion data modeling analysis sub-module: used for calling a pre-trained machine learning model for analysis processing based on the structured storage fusion data, to generate preliminary data fusion analysis results; Analysis result integration processing sub-module: used for integration and optimization processing of the preliminary data fusion analysis results, to remove redundant information and supplement association descriptions, to generate cross-domain data fusion analysis results.

8. The apparatus of claim 7, wherein, The fusion data structured import sub-module specifically includes the following three components: Data warehouse table structure adaptation component: used for comparing the field structure of the cross-domain fusion data set with the table structure of the target data warehouse, to generate a field mapping relationship table; Structured data conversion component: used for structured conversion processing of the cross-domain fusion data set based on the field mapping relationship table, to generate structured import-ready data; Data warehouse write component: used for calling a data warehouse write interface, to write structured import-ready data into the corresponding table according to a pre-set partitioning strategy, to generate structured storage fusion data.

9. The apparatus of claim 7, wherein, The fusion data modeling analysis sub-module specifically includes the following three components: Model selection adaptation component: used for selecting an adaptive pre-trained machine learning model from a pre-set model library and configuring model input parameters according to the analysis requirement description of the structured storage fusion data, to generate model configuration information; Model data input component: used for feature extraction and format conversion processing of the structured storage fusion data based on the model configuration information, to generate model input data; Model analysis execution component: used for calling a pre-trained machine learning model to perform analysis processing based on the model configuration information and the model input data, to output model analysis results, to generate preliminary data fusion analysis results.

10. The apparatus of claim 1, wherein, The fusion analysis result output encapsulation module specifically includes the following three sub-modules: The analysis result visualization generation submodule is configured to perform visualization processing on the cross-domain data fusion analysis result to generate a visualized analysis result. The API interface packaging submodule is configured to perform API interface packaging processing on the visualized analysis result, provide an interface for external system calling, and generate a callable analysis result API interface. The interactive result adaptation processing submodule is configured to adapt the callable analysis result API interface to a description of interactive requirements of different external systems, and generate an interactive cross-domain data fusion analysis output result.

Images