A warehouse heterogeneous data intelligent management method, device and equipment and storage medium

Abstract

The present application relates to the technical field of data processing, in particular to a warehouse heterogeneous data intelligent management method, device, equipment and computer storage medium. The present application obtains unstructured text data and structured time series data locally at each federal node, maps the data quality correction parameters to a unified dimension numerical vector; constructs a distributed clustering objective function based on adaptive weights for local clustering calculation, generates local clustering statistical information and encrypts it; receives the encrypted statistical information, calculates the cross-node cluster similarity based on the intra-cluster data consistency coefficient and the dynamic threshold, and executes merging on the clusters that meet the conditions to generate the global clustering result; updates the standard data dictionary according to the global clustering result and synchronizes it to each node, and simultaneously performs incremental update on the feature mapping parameters and clustering parameters based on the clustering effect feedback using transfer learning. The present application improves the accuracy and consistency of heterogeneous data management while ensuring data privacy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, equipment, and computer storage medium for intelligent management of heterogeneous data in warehousing. Background Technology

[0002] In the power industry supply chain warehousing sector, with the deepening of digital transformation, existing technologies typically employ a collaborative approach involving data platforms, intelligent sensing terminals, and physical traceability coding. This centralized data aggregation and processing model collects, standardizes, maps, and predicts time series data from multiple sources to achieve visualized management of the entire warehousing lifecycle. However, when processing cross-entity data, the current centralized aggregation and processing model fails to resolve the conflict between fragmented cross-entity data standards and privacy protection. Data security concerns make it difficult for federated nodes to break down "data silos," resulting in low utilization of multimodal data (such as unstructured maintenance text and structured sensor time series data) and weak heterogeneous feature fusion capabilities. Furthermore, the lack of unified coding standards easily leads to phenomena such as "one item, multiple codes" and "different records for the same fault," severely impacting supply chain collaboration efficiency and increasing operational costs. Simultaneously, existing models lack adaptive evolution mechanisms, making it difficult to achieve continuous optimization in response to dynamic changes in business scenarios, thus hindering the deep release of data value. Summary of the Invention

[0003] Therefore, the technical problem to be solved by the present invention is to overcome the problems of low utilization rate of multimodal heterogeneous data fusion, poor clustering accuracy, insufficient system real-time performance and low business value conversion rate caused by the fragmentation of cross-subject data standards and the contradiction of privacy protection in the prior art.

[0004] To address the aforementioned technical problems, this invention provides an intelligent governance method for heterogeneous data in warehousing, comprising:

[0005] Local repository data containing unstructured text data and structured time-series data is acquired locally at each federation node, and the unstructured text data and the structured time-series data are mapped into numerical vectors of a unified dimension based on data quality correction parameters. A distributed clustering objective function is constructed locally on each federated node based on adaptive weights. The numerical vector is used to perform local clustering calculations, generate local clustering statistics, and encrypt the local clustering statistics. It receives encrypted local clustering statistics uploaded by each federation node, calculates cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and performs merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result. The standard data dictionary is mapped and updated based on the global unified clustering results, and the updated standard data dictionary is synchronized to each federated node to guide subsequent local data processing. At the same time, the feature mapping parameters and clustering parameters are incrementally updated based on the clustering effect feedback using a transfer learning mechanism.

[0006] Preferably, the step of acquiring local repository data containing unstructured text data and structured time-series data locally at each federated node, and mapping the unstructured text data and the structured time-series data into numerical vectors of a unified dimension based on data quality correction parameters includes: Local warehouse data from each federation node is collected through the built-in database direct connection adapter module or standard API interface. The collected unstructured text data is cleaned to obtain cleaned text data, and the collected structured time-series data is standardized to obtain standardized time-series data. The cleaned text data is weighted and mapped based on the pre-trained word vector weight matrix and combined with the text quality correction coefficient to generate a first feature vector. At the same time, the time-series feature extraction network is used to extract time-series features from the standardized time-series data and combined with the data reliability coefficient to generate a second feature vector. The generated first feature vector and second feature vector are concatenated to obtain a numerical vector with a unified dimension.

[0007] Preferably, the step of constructing a distributed clustering objective function based on adaptive weights locally on each federated node, performing local clustering calculations using the numerical vectors, generating local clustering statistics, and encrypting the local clustering statistics includes: The adaptive weights are dynamically adjusted based on the current clustering iteration round and the distribution of data within the cluster. The adaptive weights are used to balance the contributions of the first view data corresponding to the first feature vector and the second view data corresponding to the second feature vector in the clustering process. The distance between the numerical vector and each cluster point is calculated based on the weighted distance, and then substituted into the federated distributed clustering objective function containing the adaptive weights for optimization to determine the local clustering result; The cluster particles, number of samples within a cluster, and variance within a cluster are extracted from the local clustering results as local clustering statistics. The local clustering statistics are then encrypted using a symmetric encryption algorithm to obtain encrypted local clustering statistics.

[0008] Preferably, the step of substituting the adaptive weights into the federated distributed clustering objective function for optimization to determine the local clustering result includes: A federated distributed clustering objective function is constructed as follows: minimizing the sum of weighted distances from samples within each cluster to their corresponding cluster centroids, wherein the weighted distances are obtained by weighting the text feature distances and sensor feature distances with the adaptive weights; The objective function of the federated distributed clustering is iteratively optimized using the gradient descent method until the objective function converges or reaches the preset number of iterations, thereby determining the local clustering result containing multiple clusters.

[0009] Preferably, the step of receiving encrypted local clustering statistics uploaded by each federated node, calculating cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and performing a merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result includes: Receive encrypted local clustering statistics uploaded by each federated node and perform decryption operation to obtain the cluster mass points, intra-cluster sample number and intra-cluster variance of each node; calculate text view similarity and sensor view similarity based on the cluster mass points of the two clusters to be merged, and calculate inter-cluster similarity by combining adaptive weights and intra-cluster data consistency coefficient, wherein the intra-cluster data consistency coefficient is calculated based on intra-cluster sample variance; A dynamic threshold is set. When the calculated inter-cluster similarity is greater than or equal to the dynamic threshold, a cluster merging operation is triggered to merge the two clusters into a new cluster and update the global cluster structure. This process is repeated until there are no cluster pairs that meet the merging conditions to generate a globally unified clustering result.

[0010] Preferably, the calculation of inter-cluster similarity by combining adaptive weights and intra-cluster data consistency coefficients includes: The similarity between two clusters to be fused is calculated using the cluster fusion similarity formula, which is expressed as follows: the text centroid cosine similarity and sensor centroid cosine similarity of the two clusters are weighted and averaged based on the adaptive weights, and then multiplied by the intra-cluster data consistency coefficient calculated based on the intra-cluster sample variance. The calculated similarity is used as the basis for determining whether to perform a cluster merging operation.

[0011] Preferably, the step of simultaneously updating the feature mapping parameters and clustering parameters incrementally using a transfer learning mechanism based on clustering effect feedback includes: The clustering performance indicators, including clustering accuracy and cluster structure stability, are summarized periodically from feedback from each federated node. The correlation between the clustering performance indicators and the current model parameters is analyzed using a federated transfer learning algorithm. High-confidence samples are extracted from the unlabeled local clustering results as pseudo-label data. Based on the pseudo-label data, incremental update operations are performed on the feature mapping parameters and clustering parameters to generate updated feature mapping parameters and clustering parameters. The updated feature mapping parameters and clustering parameters are encrypted and distributed to each federation node to replace the old model parameters on each federation node to achieve adaptive model evolution.

[0012] The present invention also provides an intelligent management device for heterogeneous data in warehousing, comprising: The data acquisition and mapping module is used to acquire local warehouse data containing unstructured text data and structured time-series data at each federation node, and map the unstructured text data and the structured time-series data into numerical vectors of a unified dimension based on data quality correction parameters. The local clustering and encryption module is used to construct a distributed clustering objective function based on adaptive weights on each federated node, perform local clustering calculations using the numerical vectors, generate local clustering statistics, and encrypt the local clustering statistics. The global fusion module is used to receive encrypted local clustering statistics uploaded by each federated node, calculate cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and perform merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result. The update and synchronization module is used to map and update the standard data dictionary according to the global unified clustering results, and synchronize the updated standard data dictionary to each federated node to guide subsequent local data processing. At the same time, based on the clustering effect feedback, the module uses a transfer learning mechanism to incrementally update the feature mapping parameters and clustering parameters.

[0013] This invention also provides an intelligent management device for heterogeneous data in warehousing, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the above-described intelligent governance method for heterogeneous data in warehouses when executing the computer program.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described intelligent governance method for heterogeneous data in warehousing.

[0015] The technical solution of the present invention has the following advantages over the prior art: The intelligent governance method for heterogeneous data in warehousing described in this invention maps unstructured text data and structured time-series data into numerical vectors of a unified dimension locally at each federated node. It then constructs a distributed clustering objective function based on adaptive weights for local clustering calculation and encryption. Next, at the coordination layer, it calculates cross-node cluster similarity and selectively merges clusters based on intra-cluster data consistency coefficients and dynamic thresholds to generate a globally unified clustering result. Finally, it updates the standard data dictionary based on the global clustering result and synchronizes it to each node. Simultaneously, it utilizes a transfer learning mechanism to incrementally update feature mapping parameters and clustering parameters, combined with clustering effect feedback. This significantly improves the clustering accuracy of cross-entity heterogeneous data, the system's real-time processing capability, and the business value conversion rate, while ensuring that the original data remains within its domain and effectively resolving the contradiction between privacy protection and heterogeneous data fusion. Attached Figure Description

[0016] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the implementation of an intelligent governance method for heterogeneous data in warehousing provided by the present invention. Figure 2 This is a structural block diagram of a warehouse heterogeneous data intelligent management device provided in an embodiment of the present invention. Detailed Implementation

[0017] The core of this invention is to provide a method, apparatus, equipment, and computer storage medium for intelligent governance of heterogeneous data in warehousing, so as to effectively solve the problems of low utilization rate of multimodal heterogeneous data fusion, poor clustering accuracy, and low business value conversion rate caused by the fragmentation of cross-entity data standards and the contradiction between privacy protection in the prior art.

[0018] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please refer to Figure 1. Figure 1 The flowchart illustrates the implementation of an intelligent governance method for heterogeneous data in warehousing provided by this invention; the specific operation steps are as follows: S101: Local repository data containing unstructured text data and structured time-series data is obtained locally at each federation node, and the unstructured text data and the structured time-series data are mapped into numerical vectors of a unified dimension based on data quality correction parameters. S102: Construct a distributed clustering objective function locally on each federated node based on adaptive weights, perform local clustering calculations using the numerical vectors, generate local clustering statistics, and encrypt the local clustering statistics. S103: Receive encrypted local clustering statistics uploaded by each federation node, calculate cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and perform merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result. S104: The standard data dictionary is mapped and updated according to the global unified clustering results, and the updated standard data dictionary is synchronized to each federated node to guide subsequent local data processing. At the same time, the feature mapping parameters and clustering parameters are incrementally updated based on the clustering effect feedback using a transfer learning mechanism.

[0020] In some embodiments, regarding the acquisition of local warehouse data and dual-view feature mapping process in step S101, the local warehouse data of each federation node is first collected as input source through the built-in database direct connection adapter module or standard API interface. For older WMS systems deployed before 2015 without standard API interfaces, the built-in database direct connection adapter module is enabled to directly connect to Oracle or InfluxDB databases to extract structured business data and high-frequency barcode scanning data. For new WMS systems with standard interfaces, multimodal data is acquired in batches through standardized API interfaces, and the MQTT and TCP / IP protocol automatic conversion engine is used to resolve data protocol conflicts between different manufacturers' devices, outputting a continuous and uninterrupted raw multimodal data stream. Subsequently, the processing is divided into two parallel paths: for unstructured maintenance text data, word segmentation, deduplication, and professional terminology unification processing are performed to obtain cleaned maintenance text data; for structured sensor time-series data, outlier removal and standardization processing are performed based on the z-score normalization principle to obtain standardized sensor data.

[0021] In one specific embodiment, in the sub-step of generating a unified-dimensional numerical vector, the input sources are the cleaned maintenance text data and standardized sensor data mentioned above. Specifically, the TF-IDF value is calculated for each word in the cleaned maintenance text data, and an initial text feature matrix is ​​constructed. A pre-trained word vector weight matrix covering more than 1200 material types and more than 500 equipment fault types in the power warehousing domain is then used to perform a weighted transformation on this initial text feature matrix. A text quality correction coefficient λ, ranging from 0.8 to 1.0, is dynamically determined based on the ambiguity and abbreviation of the maintenance records, according to the formula... The weighted feature matrix is ​​multiplied by the text quality correction coefficient to obtain the corrected text feature representation, which is then subjected to dimensionality reduction projection to generate a first feature vector with a dimension of 256. Simultaneously, multidimensional standardized sensor data containing temperature, vibration, and voltage parameters are input into an LSTM network for forward propagation calculation, and the output is nonlinearly transformed using the ReLU activation function to obtain the initial temporal features.

[0022] The data reliability coefficient r is dynamically generated based on the sensor calibration status and transmission delay, according to the formula. The initial time-series features are multiplied by the data reliability coefficient to obtain weighted time-series features, which are then subjected to a linear mapping operation to generate a second feature vector with a dimension of 256. Finally, the generated first and second feature vectors are concatenated to output a numerical vector of uniform dimension 256 for subsequent local clustering calculations.

[0023] It should be noted that this implementation method achieves seamless data access between old and new systems through dual-chip adaptation and automatic protocol conversion. Combined with domain-specific word vector matrices and dynamic correction coefficients, it effectively eliminates recording errors of unstructured text and noise interference from time-series data, significantly improving the accuracy and consistency of heterogeneous data feature representation, and providing high-quality unified dimensional input for subsequent federated clustering.

[0024] In other embodiments, the specific implementation process of security protection for mobile scanning terminals and material tags during the data acquisition phase is as follows. First, regarding the input source of material identification, the system uses a dual-chip encrypted tag integrating QR code and NFC chip as the physical carrier. This tag has built-in dynamic anti-counterfeiting code generation logic. The processing action is to control the security chip inside the tag to perform a dynamic anti-counterfeiting code refresh operation that is updated once per hour, using a timestamp and encryption seed to generate a new verification code. The output result is a material identification that is time-sensitive and difficult to counterfeit, effectively preventing the tag from being statically copied. Second, regarding the data transmission and access input of the mobile scanning terminal, after the terminal collects the tag information, the processing action is to forcibly enable the AES-256 encryption transmission protocol to encapsulate the data, and to execute a strong password verification strategy when the user logs in or operates. This strategy requires the input password to be at least 8 characters in complexity and must contain letters, numbers, and special characters simultaneously. If the verification fails, the connection is refused. The output result is a highly encrypted transmission data stream and a legitimate terminal session that has passed identity authentication, eliminating the security risks caused by weak passwords. Finally, for data access requests from supplier nodes, the system performs permission determination based on the RBAC (Role-Based Access Control) model. The process involves reading the pre-configured role permission table, matching the supplier node's role identifier with the requested data resource, allowing only non-sensitive data query commands within the authorized scope, and blocking any requests attempting to obtain raw warehouse data. The output is a finely filtered data access response, ensuring that suppliers can only access anonymized data necessary for their business and cannot access core raw data. Through these progressively layered security processes, a complete closed loop is constructed, from physical tag anti-counterfeiting and transmission link encryption to application-layer permission control.

[0025] It should be noted that this implementation method, through the combination of dual-chip dynamic anti-counterfeiting and high-strength encryption verification, significantly improves the authenticity of the data source in the warehouse and the security of the transmission process. It effectively solves the risks of data leakage caused by the easy counterfeiting of traditional QR codes and weak terminal passwords. At the same time, the fine-grained access control based on the RBAC model realizes privacy protection in cross-entity data sharing, which complies with data security regulations.

[0026] Based on the above embodiments, in some embodiments, local repository data containing unstructured text data and structured time-series data is acquired locally at each federated node, and the unstructured text data and the structured time-series data are mapped into numerical vectors of a unified dimension based on data quality correction parameters, including: Local warehouse data from each federation node is collected through the built-in database direct connection adapter module or standard API interface. The collected unstructured text data is cleaned to obtain cleaned text data, and the collected structured time-series data is standardized to obtain standardized time-series data. The cleaned text data is weighted and mapped based on the pre-trained word vector weight matrix and combined with the text quality correction coefficient to generate a first feature vector. At the same time, the time-series feature extraction network is used to extract time-series features from the standardized time-series data and combined with the data reliability coefficient to generate a second feature vector. The generated first feature vector and second feature vector are concatenated to obtain a numerical vector with a unified dimension.

[0027] In one specific embodiment, for unstructured maintenance text data, word segmentation, deduplication, and professional terminology standardization are performed to obtain cleaned maintenance text data. For structured sensor time-series data, outlier removal and standardization are performed based on the z-score normalization principle to obtain standardized sensor data. A TF-IDF value is calculated for each word in the cleaned maintenance text data to construct an initial text feature matrix. A weighted transformation is performed on this initial text feature matrix using a pre-trained word vector weight matrix covering over 1200 material types and over 500 equipment fault types in the power warehousing domain. A text quality correction coefficient λ, ranging from 0.8 to 1.0, is dynamically determined based on the ambiguity and abbreviation of the maintenance records. The weighted feature matrix is ​​multiplied by the text quality correction coefficient to obtain the corrected text feature representation, which is then subjected to dimensionality reduction projection to generate a first feature vector with dimension 256. Simultaneously, multi-dimensional standardized sensor data containing temperature, vibration, and voltage indicators are input into an LSTM network for forward propagation calculation, and the output result is nonlinearly transformed using the ReLU activation function to obtain the initial time-series features. The data reliability coefficient r is dynamically generated based on the sensor calibration status and transmission delay. The initial time series features are multiplied by the data reliability coefficient to obtain the weighted time series features, and then a second feature vector with a dimension of 256 is generated through a linear mapping operation.

[0028] It should be noted that by introducing data quality correction parameters to perform weighted mapping on heterogeneous data, this step can effectively shield the recording errors and signal noise of a single data source, significantly improve the robustness and accuracy of feature representation, and enable subsequent cluster analysis to more accurately identify complex relationships such as "different records with the same fault" or "one item with multiple codes", thus ensuring the quality and efficiency of cross-subject data fusion from the source.

[0029] Based on the above embodiments, in some embodiments, constructing a distributed clustering objective function locally on each federated node based on adaptive weights, performing local clustering calculations using the numerical vectors, generating local clustering statistics, and encrypting the local clustering statistics include: The adaptive weights are dynamically adjusted based on the current clustering iteration round and the distribution of data within the cluster. The adaptive weights are used to balance the contributions of the first view data corresponding to the first feature vector and the second view data corresponding to the second feature vector in the clustering process. The distance between the numerical vector and each cluster point is calculated based on the weighted distance, and then substituted into the federated distributed clustering objective function containing the adaptive weights for optimization to determine the local clustering result; The cluster particles, number of samples within a cluster, and variance within a cluster are extracted from the local clustering results as local clustering statistics. The local clustering statistics are then encrypted using a symmetric encryption algorithm to obtain encrypted local clustering statistics.

[0030] In one specific embodiment, the federated node first receives a unified-dimensional numerical vector generated by dual-view feature mapping as input. This numerical vector contains text feature vectors corresponding to the first-view data and sensor feature vectors corresponding to the second-view data. In terms of processing, the node monitors the current clustering iteration 't' and the data distribution within the cluster, dynamically adjusting the adaptive weight α(t) accordingly. Specifically, if high consistency of the maintenance text data is detected, the weight value corresponding to the first-view data is increased to the range of 0.6 to 0.7; if high integrity of the sensor time-series data is detected, the weight value corresponding to the second-view data is decreased, thus adjusting the weight value corresponding to the first-view data accordingly, thereby ensuring dynamic matching between the dual-view feature contribution and data quality. Subsequently, the node calculates the distance between each numerical vector and each cluster point based on weighted Euclidean distance, with the distance calculation process assigning higher weights to fault-related sensor indicators and core maintenance fields.

[0031] Specifically, the calculated distances are substituted into the federated distributed clustering objective function for optimization. This objective function is constructed as follows:

[0032] in and Let be the text feature vector and the sensor feature vector of the i-th sample, respectively. and The text centroid and sensor centroid of the k-th cluster are respectively. The objective function is iteratively optimized using gradient descent until it converges or reaches a preset number of iterations, thus determining the local clustering result containing K clusters as the output. Based on the determined local clustering result, the system further extracts cluster centroids, the number of samples within each cluster, and the intra-cluster variance as local clustering statistics. Finally, the above local clustering statistics are encrypted using the AES-256 encryption algorithm to generate encrypted local clustering statistics for secure uploading to the federated coordination layer.

[0033] It should be noted that this implementation method, by dynamically adjusting adaptive weights, accurately matches the contribution of dual-view features with real-time data quality, effectively solving the clustering bias problem caused by differences in the importance of data sources in different warehousing scenarios, and improving the clustering accuracy by more than 15% compared to a single view. Simultaneously, by combining weighted Euclidean distance and AES-256 encryption, it ensures that key features are given priority while guaranteeing that the original data does not leave the domain and that statistical information is transmitted securely, thus eliminating the risk of privacy leakage at its source.

[0034] Based on the above embodiments, in some embodiments, substituting the federated distributed clustering objective function containing the adaptive weights for optimization to determine the local clustering result includes: A federated distributed clustering objective function is constructed as follows: minimizing the sum of weighted distances from samples within each cluster to their corresponding cluster centroids, wherein the weighted distances are obtained by weighting the text feature distances and sensor feature distances with the adaptive weights; The objective function of the federated distributed clustering is iteratively optimized using the gradient descent method until the objective function converges or reaches the preset number of iterations, thereby determining the local clustering result containing multiple clusters.

[0035] In some embodiments, including: In one specific embodiment, the constructed federated distributed clustering objective function takes the following form:

[0036] in, This represents a weighted Euclidean distance, which assigns higher weights to fault-related sensor indicators and core maintenance fields.

[0037] It should be noted that by iteratively optimizing the objective function using gradient descent, the optimal local clustering result can be adaptively determined, providing a high-quality statistical information basis for subsequent global fusion.

[0038] Based on the above embodiments, in some embodiments, receiving encrypted local clustering statistics uploaded by each federated node, calculating cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and performing a merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result includes: Receive encrypted local clustering statistics uploaded by each federated node and perform decryption operation to obtain the cluster mass points, intra-cluster sample number and intra-cluster variance of each node; calculate text view similarity and sensor view similarity based on the cluster mass points of the two clusters to be merged, and calculate inter-cluster similarity by combining adaptive weights and intra-cluster data consistency coefficient, wherein the intra-cluster data consistency coefficient is calculated based on intra-cluster sample variance; A dynamic threshold is set. When the calculated inter-cluster similarity is greater than or equal to the dynamic threshold, a cluster merging operation is triggered to merge the two clusters into a new cluster and update the global cluster structure. This process is repeated until there are no cluster pairs that meet the merging conditions to generate a globally unified clustering result.

[0039] In one specific embodiment, the federated coordination layer first receives local clustering statistics encrypted with the AES-256 algorithm uploaded by each federated node. Through decryption, it reconstructs the local cluster centroids, intra-cluster sample counts, and intra-cluster variance for each node. The cluster centroids include both text view centroids and sensor view centroids. Subsequently, the system selects the m-th and n-th clusters to be fused as processing objects and extracts their text view centroids. and sensor view centroid Calculate the cosine similarity between the text and the view respectively. Cosine similarity to sensor view The system compares the calculated inter-cluster similarity with a preset dynamic threshold, which is set to range from 0.8 to 0.9 and can be adaptively adjusted according to the cross-subject data similarity distribution.

[0040] It should be noted that this specific implementation method effectively avoids the problem of erroneous merging caused by the discreteness of data within a single cluster or the fixed threshold by introducing dual constraints of intra-cluster data consistency coefficient and dynamic threshold. It significantly improves the accuracy and consistency of cross-subject heterogeneous data clustering, thereby solving the data governance problems caused by "one item, multiple codes" and "different records for the same fault".

[0041] Based on the above embodiments, in some embodiments, calculating inter-cluster similarity by combining adaptive weights and intra-cluster data consistency coefficients includes: The similarity between two clusters to be fused is calculated using the cluster fusion similarity formula, which is expressed as follows: the text centroid cosine similarity and sensor centroid cosine similarity of the two clusters are weighted and averaged based on the adaptive weights, and then multiplied by the intra-cluster data consistency coefficient calculated based on the intra-cluster sample variance. The calculated similarity is used as the basis for determining whether to perform a cluster merging operation.

[0042] In one specific embodiment, the similarity between the m-th cluster and the n-th cluster to be fused is calculated using the following cluster fusion similarity formula:

[0043] in, The adaptive weights for the current clustering iteration round. Let cosine similarity be the centroids of the two clusters of texts. Let cosine similarity be the centroids of the two cluster sensors. This is the intra-cluster data consistency coefficient calculated based on the intra-cluster sample variance. The logic for the value of is that the smaller the variance of the samples within a cluster, the closer the coefficient is to 1, which is used to suppress the erroneous merging of clusters with large dispersion.

[0044] It should be noted that this formula achieves accurate quantification of cross-node cluster similarity by integrating dual-view similarity and consistency coefficient, providing a reliable basis for dynamic threshold determination.

[0045] Based on the above embodiments, in some embodiments, incremental updates of feature mapping parameters and clustering parameters are performed simultaneously using a transfer learning mechanism based on clustering effect feedback, including: The clustering performance indicators, including clustering accuracy and cluster structure stability, are summarized periodically from feedback from each federated node. The correlation between the clustering performance indicators and the current model parameters is analyzed using a federated transfer learning algorithm. High-confidence samples are extracted from the unlabeled local clustering results as pseudo-label data. Based on the pseudo-label data, incremental update operations are performed on the feature mapping parameters and clustering parameters to generate updated feature mapping parameters and clustering parameters. The updated feature mapping parameters and clustering parameters are encrypted and distributed to each federation node to replace the old model parameters on each federation node to achieve adaptive model evolution.

[0046] In one specific embodiment, the federated coordination layer summarizes the clustering performance indicators, including clustering accuracy and cluster structure stability, federated at each federated node weekly. A federated transfer learning algorithm is then used to analyze the correlation between these clustering performance indicators and the current model parameters. High-confidence samples are extracted from the unlabeled local clustering results as pseudo-label data. The word vector weight matrix is ​​then adjusted based on this pseudo-label data. The text quality correction coefficient λ, data reliability coefficient r, and adaptive weight initialization parameters are incrementally updated to generate updated feature mapping parameters and clustering parameters. These updated feature mapping parameters and clustering parameters are then encrypted and distributed to each federated node, replacing the old model parameters locally on each federated node to achieve adaptive model evolution.

[0047] Specifically, the federated coordination layer, as the input source, periodically receives and aggregates clustering performance metrics uploaded from various federated nodes. The processing involves invoking a federated transfer learning algorithm to construct a mapping relationship between the clustering performance metrics and the current model parameters, analyzing the weight of parameter bias on the governance effect. The output is a parameter adjustment strategy vector, clarifying the direction and magnitude of subsequent updates. Next, samples with confidence levels higher than a preset threshold are selected from the local clustering results as pseudo-label data. This pseudo-label data is used to optimize the word vector weight matrix W_emb, text quality correction coefficient λ, data reliability coefficient r, and adaptive weight initialization parameters using gradient descent. Subsequently, the parameter set is encrypted using the AES-256 encryption algorithm and distributed to each federated node through a secure channel. Each node decrypts the encrypted data and directly replaces the old parameters in its local feature extraction and clustering models. Finally, the updated model is immediately put into the next round of local data processing, forming a closed-loop logic from performance feedback to parameter optimization, achieving continuous improvement in model accuracy without manual labeling of training data.

[0048] It should be noted that this implementation method, by introducing federated transfer learning and pseudo-labeling mechanisms, achieves automated incremental updates of model parameters, reducing model adaptation latency from 20 person-days of manual labeling to less than half a day, significantly reducing operation and maintenance costs. At the same time, iterative optimization using high-confidence unlabeled data effectively solves the model lag problem caused by dynamic changes in fault types in power storage scenarios, ensuring the real-time performance and accuracy of data governance.

[0049] In one specific embodiment, this embodiment will describe in detail the complete implementation of a smart governance method for heterogeneous data in warehousing. This method is based on the Federated Dual-View Clustering Fusion Algorithm (Fed-DVCF) and aims to solve technical problems such as the fragmentation of cross-entity data standards, weak multimodal data fusion, and the imbalance between data security and real-time processing efficiency in the power industry supply chain warehousing.

[0050] In this embodiment, the method operates on a layered federated learning architecture, which, from bottom to top, includes a terminal layer, a federated node layer, and a federated coordination layer. The terminal layer covers various warehouse IoT devices and different versions of Warehouse Management Systems (WMS), specifically including sensors, RFID dual-chip terminals, 4K cameras, and WMS systems deployed in provincial, municipal, and county-level warehouses. For some county-level warehouses with older WMS systems deployed before 2015 that lack standard API interfaces, the terminal layer incorporates a built-in database direct connection adaptation module. This module can directly connect to Oracle or InfluxDB databases, extract structured business data and high-frequency barcode scanning data, and convert them to standard formats using a data format conversion tool, thereby improving data synchronization stability to over 99%. For new WMS systems and various IoT devices with standard interfaces, the terminal layer collects data through standardized API interfaces or an MQTT / TCP / IP protocol automatic conversion engine. It supports access to more than eight heterogeneous formats, including JSON, XML, CSV, text, images, and time-series streaming data. This eliminates the need for additional gateway development, resolving data protocol conflicts between devices from different manufacturers and achieving continuous data collection with zero monthly data interruptions. The collected local warehouse data mainly includes two types: the first view data is unstructured maintenance text records, and the second view data is structured equipment sensor time series data, which includes core indicators such as temperature, vibration, and voltage.

[0051] Before the data enters the feature extraction stage, each federated node performs a rigorous data preprocessing procedure locally. For unstructured maintenance text data, the system first performs cleaning operations, removing invalid characters and performing word segmentation; then, it calls a professional terminology library in the power storage field to unify the vocabulary and eliminate synonym ambiguity; finally, outliers are removed based on the 3σ principle. For structured sensor time-series data, the system uses the z-score normalization principle for standardization processing to eliminate the influence of dimensions and remove abnormal data points that exceed the normal fluctuation range. In addition, for the collected unstructured image data (such as equipment failure photos), this embodiment uses the YOLOv8 algorithm to parse key features and integrates the parsing results into the sensor data view to further improve feature richness. After preprocessing, the cleaned maintenance text data and standardized sensor data are obtained, preparing for subsequent dual-view feature mapping.

[0052] Next, each federated node performs dual-view feature mapping locally, converting heterogeneous data into numerical vectors of a unified dimension. Specifically, for the cleaned maintenance text data, the system performs a weighted transformation based on a word vector weight matrix pre-trained from a corpus in the power storage domain. This matrix covers more than 1200 material types and more than 500 equipment fault types. The system calculates the TF-IDF value for each word in the text and constructs an initial text feature matrix, which is then multiplied by the word vector weight matrix to obtain the weighted transformed feature representation. To address potential fuzzy or abbreviated records in maintenance logs, this embodiment introduces a text quality correction coefficient λ, which is dynamically determined based on the clarity and completeness of the record, ranging from 0.8 to 1.0. The weighted transformed feature matrix is ​​multiplied by the text quality correction coefficient λ to obtain the corrected text feature representation. Finally, a dimension reduction projection operation is performed to generate a feature representation with dimension λ. First eigenvector This process can be represented as: Where T represents the cleaned and deduplicated unstructured maintenance text. This is the word vector weight matrix.

[0053] Simultaneously, for standardized sensor data, the system utilizes a Long Short-Term Memory (LSTM) network for temporal feature extraction. Multidimensional time-series data containing temperature, vibration, and voltage parameters are input into the LSTM network for forward propagation calculation. The output is then nonlinearly transformed using the ReLU activation function σ to obtain initial temporal features. Considering potential calibration biases or transmission delays in sensors, this embodiment introduces a data reliability coefficient r, which is dynamically assigned based on the sensor calibration status and transmission delay. The initial temporal features are multiplied by the data reliability coefficient r to obtain weighted temporal features, which are then subjected to a linear mapping operation to generate a dimension of... The second eigenvector This process can be represented as ,in For standardized multidimensional time-series sensor data, and These are the weight matrix and the bias term, respectively. Finally, the generated first and second eigenvectors are concatenated to obtain a numerical vector with a unified dimension of 256, thus achieving a unified representation of heterogeneous data in the feature space.

[0054] After feature mapping is completed, each federated node constructs a federated distributed clustering objective function based on adaptive weights and performs local clustering calculations using the generated numerical vectors. This embodiment introduces an adaptive weight α(t), which is dynamically adjusted based on the current clustering iteration t and the data distribution within the cluster. Specifically, the initial value is set to 0.5. If high consistency of the text data is detected, the weight value corresponding to the first view data is increased to 0.6 to 0.7; if high integrity of the sensor data is detected, the weight value corresponding to the second view data is decreased to 0.3 to 0.4. The system calculates the distance between the numerical vector and each cluster point based on weighted Euclidean distance, assigning higher weights to fault-related sensor indicators and core maintenance fields to improve clustering accuracy. The local clustering calculation aims to optimize the following federated distributed clustering objective function:

[0055] After local clustering is completed, each federated node extracts key statistical information from the local clustering results, including cluster particles, number of samples within a cluster, and intra-cluster variance, as local clustering statistics. To ensure privacy and security during data transmission, each node encrypts this local clustering statistics using the AES-256 encryption algorithm, obtaining encrypted local clustering statistics, which are then uploaded to the federated coordination layer. Throughout this process, the original repository data remains locally on each federated node, with only non-sensitive statistical features uploaded, fundamentally resolving the contradiction between "data silos" and privacy leaks.

[0056] After receiving the AES-256 encrypted local clustering statistics uploaded by each federation node, the federation coordination layer first performs a decryption operation to obtain the cluster data points, intra-cluster sample count, and intra-cluster variance for each node. Subsequently, the coordination layer performs global fusion of clusters across nodes based on cluster fusion similarity conditions. Specifically, the system calculates the similarity between the two clusters to be fused (denoted as the m-th cluster and the n-th cluster). The formula for calculating cluster fusion similarity is as follows:

[0057] The system sets a dynamic threshold θ, ranging from 0.8 to 0.9, and adaptively adjusts it based on the cross-subject data similarity distribution. When the calculated inter-cluster similarity... When the cluster size is greater than or equal to the dynamic threshold θ, a cluster merging operation is triggered, merging the two clusters into a new cluster and updating the global cluster structure. This process is repeated until no cluster pair satisfies the merging condition, ultimately generating a globally unified clustering result. This mechanism effectively solves the clustering chaos caused by "one item, multiple codes" and "different records for the same fault," achieving a 98% consistency in cross-subject clustering results.

[0058] After generating a globally unified clustering result, the federated coordination layer connects to the core field standards of the DAMA framework to establish a cross-entity unified data dictionary. The system maps each cluster in the globally unified clustering result to the standard coding system in this unified data dictionary, identifying and merging the standard codes corresponding to "one item, multiple codes" and "same fault, different records." Subsequently, the federated coordination layer distributes the updated standard data dictionary to the WMS system, procurement system, and inventory management system of each federated node. Based on the updated standard data dictionary, each federated node performs standardized classification and storage operations on newly generated local warehouse data, thereby significantly improving the efficiency of cross-departmental data queries, reducing query time from 2 hours to 30 seconds.

[0059] In terms of security, this embodiment implements a multi-layered protection strategy. At the material identification level, a dual-chip encrypted tag integrating QR code and NFC chip is used, with a built-in dynamic anti-counterfeiting code that updates hourly, effectively preventing tag counterfeiting and keeping the annual tag damage rate below 3%. At the terminal security level, the AES-256 encrypted transmission protocol is enforced on mobile scanning terminals, and a strong password verification strategy is implemented, requiring passwords to be at least 8 characters long and containing letters, numbers, and special characters, completely eliminating the risk of weak passwords. At the access control level, fine-grained permissions are configured for supplier nodes based on the RBAC role-based access control model, restricting suppliers to accessing only non-sensitive data within the authorized scope and preventing them from obtaining original warehouse data, thus complying with data security regulations.

[0060] Furthermore, this embodiment achieves deep data value transformation through a business linkage module. The system deeply integrates clustering results with the procurement system, maintenance scheduling system, and inventory management system, constructing a closed-loop process of "fault early warning → maintenance dispatch → spare parts procurement." When the clustering algorithm identifies potential equipment failure modes, it automatically triggers the maintenance dispatch process and adjusts spare parts inventory in conjunction with the procurement system based on the failure type. This linkage mechanism increases the data value conversion rate from the current 38% to over 80%, effectively reducing warehousing labor costs and equipment failure rates. Simultaneously, the system possesses strong scenario adaptability, compatible with different data scales of large hub warehouses and small county-level warehouses. Adding new business domains does not require restructuring core modules, and the expansion cycle is shortened to within 7 days.

[0061] It should be noted that this embodiment utilizes a federated dual-view clustering fusion algorithm, combined with dual-chip dynamic anti-counterfeiting labels, legacy system adaptation modules, and a federated transfer learning mechanism, to construct a complete intelligent governance solution for heterogeneous data in warehousing. This solution not only achieves efficient fusion and standardized governance of heterogeneous data, but also significantly improves cross-domain fusion efficiency, clustering accuracy, and business value conversion rate while ensuring data privacy and security, providing solid technical support for the digital transformation of power warehousing.

[0062] Please refer to Figure 2 , Figure 2 A structural block diagram of a warehouse heterogeneous data intelligent management device provided in an embodiment of the present invention; the specific device may include: The data acquisition and mapping module 100 is used to acquire local warehouse data containing unstructured text data and structured time-series data at each federation node, and map the unstructured text data and the structured time-series data into numerical vectors of a unified dimension based on data quality correction parameters. The local clustering and encryption module 200 is used to construct a distributed clustering objective function based on adaptive weights on each federated node, perform local clustering calculations using the numerical vectors, generate local clustering statistics, and encrypt the local clustering statistics. The global fusion module 300 is used to receive encrypted local clustering statistics uploaded by each federation node, calculate cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and perform merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result. The update and synchronization module 400 is used to map and update the standard data dictionary according to the global unified clustering result, and synchronize the updated standard data dictionary to each federated node to guide subsequent local data processing. At the same time, based on the clustering effect feedback, the feature mapping parameters and clustering parameters are incrementally updated using a transfer learning mechanism.

[0063] The intelligent governance device for heterogeneous warehouse data in this embodiment is used to implement the aforementioned intelligent governance method for heterogeneous warehouse data. Therefore, the specific implementation of the intelligent governance device for heterogeneous warehouse data can be found in the previous embodiment section of the intelligent governance method for heterogeneous warehouse data. For example, the data acquisition and mapping module 100, the local clustering and encryption module 200, the global fusion module 300, and the update and synchronization module 400 are respectively used to implement steps S101, S102, S103, and S104 in the above-mentioned intelligent governance method for heterogeneous warehouse data. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0064] In one specific embodiment, the data acquisition and mapping module 100 is specifically used for: collecting local warehousing data from each federated node through a built-in database direct connection adaptation module or a standard API interface; performing word segmentation, deduplication, and professional terminology unification processing on the collected unstructured maintenance text data to obtain cleaned maintenance text data; simultaneously performing outlier removal and standardization processing on the collected structured sensor time-series data based on the z-score normalization principle to obtain standardized sensor data; performing TF-IDF weighted mapping on the cleaned maintenance text data based on the word vector weight matrix pre-trained from the power warehousing corpus and combining it with the text quality correction coefficient to generate a first feature vector; simultaneously using an LSTM network to perform time-series feature extraction on the standardized sensor data and combining it with the data reliability coefficient to generate a second feature vector; and performing a dimension concatenation operation on the generated first feature vector and second feature vector to obtain a numerical vector with a unified dimension.

[0065] In one specific embodiment, the local clustering and encryption module 200 is specifically used to: dynamically adjust adaptive weights based on the current clustering iteration round and the data distribution within the clusters; increase the weight value corresponding to the first view data when the text data consistency is high, and decrease the weight value corresponding to the second view data when the sensor data integrity is high; calculate the distance between the numerical vector and each cluster particle based on the weighted Euclidean distance, and substitute it into the federated distributed clustering objective function containing the adaptive weights for optimization to determine the local clustering result, wherein the form of the federated distributed clustering objective function is:

[0066] The cluster particles, number of samples within a cluster, and variance within a cluster are extracted from the local clustering results as local clustering statistics, and the local clustering statistics are encrypted using the AES-256 encryption algorithm.

[0067] In one specific embodiment, the global fusion module 300 is specifically used to: receive local clustering statistics uploaded by each federated node and perform decryption operations to obtain the cluster mass points, intra-cluster sample count, and intra-cluster variance of each node; calculate the text view cosine similarity and sensor view cosine similarity based on the cluster mass points of the two clusters to be fused, and calculate the inter-cluster similarity using the following formula in combination with adaptive weights and intra-cluster data consistency coefficient:

[0068] A dynamic threshold is set. When the calculated inter-cluster similarity is greater than or equal to the dynamic threshold, a cluster merging operation is triggered to merge the two clusters into a new cluster and update the global cluster structure. This process is repeated until there are no cluster pairs that meet the merging conditions to generate a globally unified clustering result.

[0069] In one specific embodiment, the update and synchronization module 400 is specifically used for: establishing a cross-subject unified data dictionary by connecting to the core field standards of the DAMA framework; mapping each cluster in the globally unified clustering result to the standard coding system in the unified data dictionary; identifying and merging the standard codes corresponding to "one item, multiple codes" and "same fault, different records"; distributing the mapped and updated standard data dictionary to the WMS system, procurement system, and inventory management system of each federation node through the federation coordination layer, so that each federation node performs standardized classification and storage operations on the newly generated local warehousing data according to the updated standard data dictionary; simultaneously, summarizing the clustering effect indicators, including clustering accuracy and cluster structure stability, federated transfer learning algorithm to analyze the correlation between the clustering effect indicators and the current model parameters; extracting high-confidence samples from the unlabeled local clustering results as pseudo-label data; performing incremental update operations on the word vector weight matrix, text quality correction coefficient, data reliability coefficient, and adaptive weight initialization parameters based on the pseudo-label data; and encrypting and distributing the updated parameters to each federation node to replace the old model parameters.

[0070] It should be noted that the specific implementation methods of each module in the above device embodiments can refer to the description of the corresponding steps in the foregoing method embodiments, and will not be repeated here. This device, through a federated dual-view clustering and fusion mechanism, effectively solves the problems of cross-subject data standard fragmentation and multimodal fusion while ensuring the privacy of the original data, and significantly improves the accuracy and consistency of heterogeneous data governance.

[0071] A specific embodiment of the present invention also provides an intelligent management device for heterogeneous data in warehousing, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described intelligent management method for heterogeneous data in warehousing.

[0072] A specific embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described intelligent governance method for heterogeneous data in warehousing.

[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for intelligent governance of heterogeneous data in warehousing, characterized in that, include: Local repository data containing unstructured text data and structured time-series data is acquired locally at each federation node, and the unstructured text data and the structured time-series data are mapped into numerical vectors of a unified dimension based on data quality correction parameters. A distributed clustering objective function is constructed locally on each federated node based on adaptive weights. The numerical vector is used to perform local clustering calculations, generate local clustering statistics, and encrypt the local clustering statistics. It receives encrypted local clustering statistics uploaded by each federation node, calculates cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and performs merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result. The standard data dictionary is mapped and updated based on the global unified clustering results, and the updated standard data dictionary is synchronized to each federated node to guide subsequent local data processing. At the same time, the feature mapping parameters and clustering parameters are incrementally updated based on the clustering effect feedback using a transfer learning mechanism.

2. The method according to claim 1, characterized in that, The step of acquiring local repository data containing unstructured text data and structured time-series data locally at each federated node, and mapping the unstructured text data and the structured time-series data into numerical vectors of a unified dimension based on data quality correction parameters includes: Local warehouse data from each federation node is collected through the built-in database direct connection adapter module or standard API interface. The collected unstructured text data is cleaned to obtain cleaned text data, and the collected structured time-series data is standardized to obtain standardized time-series data. The cleaned text data is weighted and mapped based on the pre-trained word vector weight matrix and combined with the text quality correction coefficient to generate a first feature vector. At the same time, the time-series feature extraction network is used to extract time-series features from the standardized time-series data and combined with the data reliability coefficient to generate a second feature vector. The generated first feature vector and second feature vector are concatenated to obtain a numerical vector with a unified dimension.

3. The method according to claim 2, characterized in that, The step of constructing a distributed clustering objective function based on adaptive weights locally on each federated node, performing local clustering calculations using the numerical vectors, generating local clustering statistics, and encrypting the local clustering statistics includes: The adaptive weights are dynamically adjusted based on the current clustering iteration round and the distribution of data within the cluster. The adaptive weights are used to balance the contributions of the first view data corresponding to the first feature vector and the second view data corresponding to the second feature vector in the clustering process. The distance between the numerical vector and each cluster point is calculated based on the weighted distance, and then substituted into the federated distributed clustering objective function containing the adaptive weights for optimization to determine the local clustering result; The cluster particles, number of samples within a cluster, and variance within a cluster are extracted from the local clustering results as local clustering statistics. The local clustering statistics are then encrypted using a symmetric encryption algorithm to obtain encrypted local clustering statistics.

4. The method according to claim 3, characterized in that, The optimization solution obtained by substituting the adaptive weights into the federated distributed clustering objective function to determine the local clustering result includes: A federated distributed clustering objective function is constructed as follows: minimizing the sum of weighted distances from samples within each cluster to their corresponding cluster centroids, wherein the weighted distances are obtained by weighting the text feature distances and sensor feature distances with the adaptive weights; The objective function of the federated distributed clustering is iteratively optimized using the gradient descent method until the objective function converges or reaches the preset number of iterations, thereby determining the local clustering result containing multiple clusters.

5. The method according to claim 1, characterized in that, The process of receiving encrypted local clustering statistics uploaded by each federated node, calculating cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and performing a merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result includes: Receive encrypted local clustering statistics uploaded by each federated node and perform decryption operation to obtain the cluster mass points, intra-cluster sample number and intra-cluster variance of each node; calculate text view similarity and sensor view similarity based on the cluster mass points of the two clusters to be merged, and calculate inter-cluster similarity by combining adaptive weights and intra-cluster data consistency coefficient, wherein the intra-cluster data consistency coefficient is calculated based on intra-cluster sample variance; A dynamic threshold is set. When the calculated inter-cluster similarity is greater than or equal to the dynamic threshold, a cluster merging operation is triggered to merge the two clusters into a new cluster and update the global cluster structure. This process is repeated until there are no cluster pairs that meet the merging conditions to generate a globally unified clustering result.

6. The method according to claim 5, characterized in that, The calculation of inter-cluster similarity by combining adaptive weights and intra-cluster data consistency coefficients includes: The similarity between two clusters to be fused is calculated using the cluster fusion similarity formula, which is expressed as follows: the text centroid cosine similarity and sensor centroid cosine similarity of the two clusters are weighted and averaged based on the adaptive weights, and then multiplied by the intra-cluster data consistency coefficient calculated based on the intra-cluster sample variance. The calculated similarity is used as the basis for determining whether to perform a cluster merging operation.

7. The method according to claim 1, characterized in that, The step of simultaneously updating the feature mapping parameters and clustering parameters incrementally using a transfer learning mechanism based on clustering effect feedback includes: The clustering performance indicators, including clustering accuracy and cluster structure stability, are summarized periodically from feedback from each federated node. The correlation between the clustering performance indicators and the current model parameters is analyzed using a federated transfer learning algorithm. High-confidence samples are extracted from the unlabeled local clustering results as pseudo-label data. Based on the pseudo-label data, incremental update operations are performed on the feature mapping parameters and clustering parameters to generate updated feature mapping parameters and clustering parameters. The updated feature mapping parameters and clustering parameters are encrypted and distributed to each federation node to replace the old model parameters on each federation node to achieve adaptive model evolution.

8. A smart management device for heterogeneous data in warehousing, characterized in that, include: The data acquisition and mapping module is used to acquire local warehouse data containing unstructured text data and structured time-series data at each federation node, and map the unstructured text data and the structured time-series data into numerical vectors of a unified dimension based on data quality correction parameters. The local clustering and encryption module is used to construct a distributed clustering objective function based on adaptive weights on each federated node, perform local clustering calculations using the numerical vectors, generate local clustering statistics, and encrypt the local clustering statistics. The global fusion module is used to receive encrypted local clustering statistics uploaded by each federated node, calculate cross-node cluster similarity based on intra-cluster data consistency coefficient and dynamic threshold, and perform merging operation on clusters that meet the fusion conditions to generate a globally unified clustering result. The update and synchronization module is used to map and update the standard data dictionary according to the global unified clustering results, and synchronize the updated standard data dictionary to each federated node to guide subsequent local data processing. At the same time, based on the clustering effect feedback, the module uses a transfer learning mechanism to incrementally update the feature mapping parameters and clustering parameters.

9. A smart management device for heterogeneous data in warehousing, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the intelligent governance method for heterogeneous data in warehouses as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the intelligent governance method for heterogeneous data in warehousing as described in any one of claims 1 to 7.

Images