Enterprise informatization management integration platform based on big data

Abstract

The invention provides an enterprise informatization management integration platform based on big data, which is based on Hadoop distributed computing framework construction, integrates technologies of MapReduce parallel processing, Spark real-time computing, HDFS distributed storage and the like, realizes data parallel processing through a swarm intelligence algorithm and a pheromone transmission mechanism, performs heterogeneous data mapping by utilizing a hierarchical self-encoding network, and realizes enterprise informatization management. Dynamic resource scheduling is realized in combination with a biological cell division mechanism and an entropy balance mechanism, storage management is optimized by adopting an adaptive grid partition and data temperature layering technology, and multi-scale feature fusion and knowledge reasoning support decision analysis are constructed based on Spark MLlib. According to the platform, the problems of low data processing efficiency, limited system expansion, difficulty in data sharing and the like in traditional enterprise informatization management are solved, and reliable technical support is provided for enterprise digital transformation.

Description

Technical Field

[0001] The present invention belongs to the technical field of enterprise information management, and specifically relates to an enterprise information management integration platform based on big data. Background Art

[0002] Enterprise information management is an important support for enterprise development. In recent years, with the accelerated advancement of digital transformation, enterprises have generated massive amounts of data in production, management, sales and other links. How to efficiently manage and utilize this data has become the core challenge of enterprise information construction.

[0003] In terms of data processing, it is currently difficult for enterprise information management to achieve efficient data collection, storage and analysis, and the value of data has not been fully utilized. The lack of unified data standards and management mechanisms has led to the formation of data islands between business systems, making data sharing difficult and unable to meet the needs of enterprise refined management.

[0004] In terms of system architecture, existing enterprise information management platforms generally adopt traditional architecture design, which is not scalable enough. As the scale of enterprises expands and the volume of business grows, the system performance is difficult to cope with the increasing data processing needs. In addition, the integration between systems is difficult and the maintenance cost is high, which limits the ability of enterprises to cope with rapid development.

[0005] At the business collaboration level, cross-departmental data sharing and business collaboration are inefficient. Due to the lack of a unified management view and efficient collaboration mechanism, there are serious information barriers between departments, and resources are difficult to share, which weakens the overall operational efficiency of the enterprise. At the same time, corporate decision-making lacks reliable data support, which reduces the scientificity and accuracy of management decisions.

[0006] In terms of data analysis applications, the existing system has limited data analysis capabilities. Faced with complex business scenarios, it is difficult to achieve deep data mining and value discovery, and it is difficult to provide strong support for corporate decision-making. In addition, the system has a high threshold for use, and the personalized needs of users at different levels are difficult to be effectively met.

[0007] Therefore, it is urgent to develop a new enterprise information management integration platform to solve the above problems. The platform should have the following characteristics: efficient data processing capabilities, flexible system scalability, convenient business collaboration mechanism and intelligent data analysis function. Through these characteristics, the platform will achieve efficient use of enterprise data resources, comprehensively improve the efficiency of information management, and provide strong support for the digital transformation of enterprises. Summary of the invention

[0008] The present invention provides an enterprise information management integrated platform based on big data, which is built on the Hadoop ecosystem and adopts a distributed computing and storage architecture, including the following modules:

[0009] 1. Parallel data processing hub

[0010] The heterogeneous data generated by the enterprise information system is processed based on the Hadoop distributed computing framework; the MapReduce parallel computing model is adopted, and the swarm intelligence algorithm and pheromone transmission mechanism are integrated. By building multiple data processing agent nodes, task allocation is performed based on historical pheromone concentration to achieve adaptive load balancing; adaptive wavelet compression technology is used to store enterprise historical data to achieve unified processing of structured business data, semi-structured log data and unstructured document data.

[0011] 2. Intelligent data mapping engine

[0012] Real-time processing of enterprise heterogeneous system data is achieved based on the Spark real-time computing framework; a distributed stream computing architecture is adopted, and a hierarchical autoencoder network is integrated to build a feature extraction model, and adaptive mapping of enterprise heterogeneous system data is achieved through a multi-layer encoder-decoder structure; a progressive feature selection mechanism is adopted, and data mapping rules are optimized through iterative feature screening to build an enterprise data standard system.

[0013] 3. Elastic Resource Scheduler

[0014] Based on the YARN resource scheduling framework, it adopts a microkernel architecture design; it integrates the biological cell division mechanism, abstracts computing resources into cell units, and triggers the division or merger of cell units by setting load thresholds; it integrates the entropy balance mechanism to achieve dynamic expansion of business modules, and uses the LSTM model to predict resources, supporting the pre-allocation and dynamic adjustment of computing resources.

[0015] 4. Distributed storage center

[0016] The distributed storage center uses the HDFS distributed file system and data lake architecture to build a unified data storage platform; integrates adaptive grid partitioning technology, builds grid units based on data access popularity, and realizes dynamic optimization of grid units; uses deterministic chaos mapping to optimize storage strategies, supports concurrent access by building a phase space model of storage status, and realizes hierarchical storage of hot and cold data through a data temperature stratification mechanism.

[0017] 5. Business Analysis Engine

[0018] The business analysis engine builds analysis models based on the Spark MLlib framework and integrates multi-scale feature fusion technology. It extracts and fuses business features by building a multi-layer feature extraction network and uses graph projection technology to build an enterprise operation monitoring model. It integrates a knowledge reasoning engine to support intelligent decision-making analysis of enterprise business scenarios by building a domain knowledge base and reasoning rules.

[0019] Beneficial effects of the present invention:

[0020] 1. Improve data processing efficiency

[0021] By building a data processing agent node network through swarm intelligence algorithm and pheromone transmission mechanism, we can achieve adaptive task allocation based on the big data environment, significantly improve data collection and processing efficiency, and give full play to the value of data.

[0022] 2. Enhance system expansion capabilities

[0023] Resource scheduling is performed based on the biological cell division mechanism and entropy balance mechanism. Through the adaptive division and merging operations of cell units, accurate dynamic scaling of computing resources in an enterprise information environment is achieved, solving the problem that the system is unable to cope with the needs of business growth.

[0024] 3. Optimize data sharing mechanism

[0025] The hierarchical autoencoding network combined with the progressive feature selection mechanism is used to achieve adaptive mapping of large-scale heterogeneous system data, break down data silos, improve cross-departmental data sharing and business collaboration efficiency, and build a unified enterprise data standard system.

[0026] 4. Strengthen analytical and mining capabilities

[0027] Through the innovative combination of multi-scale feature fusion technology and graph projection technology, combined with the knowledge reasoning engine, in-depth mining and analysis of enterprise operation data are achieved, providing intelligent support for enterprise management decisions.

[0028] 5. Improve storage access performance

[0029] Combining deterministic chaotic mapping with adaptive grid partitioning technology optimizes storage strategies and data distribution in big data environments, improves the performance of enterprise-level storage systems, and supports high-concurrency data access requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the architecture of the enterprise information management integration platform of the present invention is shown;

[0031] Figure 2 The flowchart of the enterprise information management integration platform described in the present invention is shown. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] Combination Figure 1 , provides an enterprise information management integration platform based on big data, including the following functional modules:

[0034] 1. Parallel data processing hub

[0035] The parallel data processing center is built on the Hadoop distributed computing framework. By integrating the improved swarm intelligence algorithm and pheromone transmission mechanism, combined with adaptive wavelet compression technology, it realizes efficient processing and intelligent storage of enterprise heterogeneous data. The center mainly processes structured business data, semi-structured log data and unstructured document data generated in the enterprise information system. Specific implementation includes:

[0036] A1). Group intelligence task allocation

[0037] In the parallel data processing center, the task allocation algorithm has been improved based on the idea of ​​swarm intelligence to meet the complexity requirements of enterprise-level tasks. In the specific implementation, the improvement of the algorithm is mainly reflected in task modeling, priority evaluation, dependency management and pheromone transmission optimization.

[0038] a). Task Modeling

[0039] In order to meet the diverse task requirements of enterprises, the parallel data processing center adopts a unified task modeling method. Each data processing task T is represented by a five-tuple (id, P, R, D, S), where:

[0040] id: unique identifier of the task, in UUID format;

[0041] P: Task priority vector, describing the urgency, resource usage, and business importance of the task;

[0042] R: Resource requirement vector, describing the computing, storage, and network resources required for the task, R = [r c ,r s ,r n ] T , where r c is the computing resource requirement, expressed in terms of the number of CPU cores and memory capacity; r s is the storage resource requirement, expressed in terms of data size; r n is the network resource demand, expressed as bandwidth demand;

[0043] D: data dependency set, describing the dependencies between tasks;

[0044] S: Task status identifier, including "pending", "processing" and "completed".

[0045] b). Priority assessment

[0046] The task priority vector P is defined as [p1,p2,p3] T , comprehensively evaluate the tasks from three dimensions: time urgency, resource occupancy, and task importance:

[0047] The time urgency p1 is calculated by normalizing the difference between the task deadline and the current time:

[0048]

[0049] in:

[0050] T d : Task deadline; T c : Current time; T m : The maximum task execution time supported by the system.

[0051] Resource occupancy p2, based on the demand for different resource types, uses a weight coefficient to balance the importance of each type of resource:

[0052] Where: w i : weight coefficient of resource type i; r i : The demand for this type of resource; R t : Total system resources.

[0053] Task importance p3 is dynamically adjusted based on task type priority and business priority:

[0054] p3=α·P t +(1-α)·P b

[0055] Where: α: weight coefficient of task type; P t : Task type priority; P b : The business priority of the task.

[0056] The above three dimensions p1, p2, and p3 meet the normalization conditions:

[0057]

[0058] c) Dependency Management

[0059] In enterprise-level data processing, the dependencies between tasks are usually complex and dynamic. For this reason, the system uses directed acyclic graph data G(V,E) to represent the dependencies between tasks, where V is the set of task nodes; E is the set of dependency edges, where edge e(i,j) means that task i must be completed before task j. By dynamically analyzing the dependencies, the system can reasonably arrange the execution order of tasks to avoid data conflicts and execution deadlocks.

[0060] d). Optimization of pheromone delivery mechanism

[0061] The dynamic update mechanism of pheromones is combined with the task processing efficiency η:

[0062] τ ij (t+1)=(1-ρ)τ ij (t)+Q·η

[0063] in:

[0064] ρ: pheromone attenuation coefficient, used to control the influence of historical information;

[0065] Q: pheromone intensity constant;

[0066] η: Task processing efficiency, calculated as:

[0067]

[0068] Where, T e ,T a are the estimated and actual processing time of the task; R e ,R a They are the estimated and actual resource consumption of the task, respectively.

[0069] When assigning tasks, the probability that processing node j is selected to execute task i is calculated as:

[0070]

[0071] in:

[0072] α, β: pheromone factor and heuristic factor respectively;

[0073] v ij : Node fitness, the calculation formula is:

[0074]

[0075] In the formula, C j is the comprehensive processing capability of the node; L j is the current load rate of the node; d ij is the transmission cost from task i to node j.

[0076] B1). Adaptive wavelet compression storage

[0077] In view of the large amount of enterprise data and the diversity of data characteristics, the parallel data processing center uses adaptive wavelet compression technology to optimize data storage efficiently. This technology achieves a balance between storage space and data characteristics through data layer decomposition, threshold compression and dynamic adjustment, ensuring the retention of important information and the compression of redundant data.

[0078] a). Wavelet decomposition

[0079] The input data sequence x(n) is first decomposed into multiple scales by discrete wavelet transform:

[0080]

[0081] in:

[0082] W ψ [j,k] represents the wavelet coefficient of the j-th layer and the k-th component, which contains the information of the input data at different frequencies and positions.

[0083] Wavelet basis function ψ j,k (t) is defined as:

[0084] ψ j,k (t) = 2 j / 2 ψ(2 j tk)

[0085] In the formula, j is the scale parameter, which indicates the level of decomposition and is used to capture the low-frequency (trend) or high-frequency (detail) components of the data; k is the translation parameter, which indicates the position on the time axis. Wavelet decomposition separates the global trend and local details in the data in a hierarchical manner. The low-frequency part represents the global trend of the data and is the key feature; the high-frequency part contains noise and detail information, which is selectively retained according to the threshold.

[0086] b). Adaptive threshold calculation

[0087] In order to remove high-frequency noise and compress redundant information, the compression threshold λ j The adaptive calculation formula is:

[0088]

[0089] in:

[0090] σ j is the standard deviation of the wavelet coefficients of the jth layer, reflecting the degree of fluctuation of the data in this layer;

[0091] N j is the number of wavelet coefficients in the jth layer, indicating the scale of the data in this layer;

[0092] σ j is the standard deviation of the j-th layer wavelet coefficients, N j is the number of coefficients of this layer.

[0093] c). Wavelet coefficient processing

[0094] After threshold calculation, the wavelet coefficient W ψ [j,k] is processed into compressed coefficients by soft threshold method

[0095]

[0096] sgn(Wψ [j,k]) is the sign function of the wavelet coefficients, retaining positive and negative information; |W ψ [j,k]|-λ j It is the valid information retained after subtracting the threshold.

[0097] Through threshold screening, the coefficient scale is compressed to reduce the redundant part of data storage. The part below the threshold is directly set to zero and regarded as noise or redundant information. For coefficients above the threshold, only moderate adjustments are made to ensure that important details are not lost; the part below the threshold is directly set to zero and regarded as noise or redundant information.

[0098] d). Dynamic compression ratio adjustment

[0099] The system dynamically adjusts the compression ratio based on the importance and storage requirements of the data:

[0100]

[0101] Where: H o ,H c are the information entropy of the original and compressed data respectively; γ is the compression coefficient.

[0102] 2. Intelligent data mapping engine

[0103] The intelligent data mapping engine is built on the Spark real-time computing framework. It integrates hierarchical autoencoding networks and progressive feature selection mechanisms to achieve real-time processing and mapping of enterprise heterogeneous system data. The engine adopts a distributed stream computing architecture to build a unified enterprise data standard system. Specific implementation includes:

[0104] A2). Hierarchical Autoencoder Network

[0105] The hierarchical autoencoder network extracts the features of heterogeneous data through a multi-layer encoder-decoder structure and completes feature mapping. The network consists of an input layer, multiple encoding layers, a feature layer, multiple decoding layers, and an output layer. Its core process includes the following three parts:

[0106] a). Coding process

[0107] For input data X∈R n (real number set vector space), deep features are gradually extracted through multi-layer encoders, the formula is as follows:

[0108] h (1) =f(W (1) X+b (1) )

[0109] in:

[0110] h (1) : Output of the first hidden layer;

[0111] W (1) : Weight matrix, randomly initialized and then learned through optimization;

[0112] b (1) : bias vector;

[0113] f(·): Activation function, using ReLU function:

[0114]

[0115] Subsequent encoding layer recursive calculation:

[0116] h (l) =f(W (l) h (l-1) +b (l) )

[0117] In the formula, l represents the layer index, h (l) 、h (l-1) is the output of the encoder layer l and layer l-1; W (l) is the weight matrix of layer l; b (l) is the bias vector of the lth layer.

[0118] Feature layer output:

[0119] Z=f(W (k) h (k-1) +b (k) )

[0120] In the formula, k represents the layer index, h (k-1) is the output of the encoder layer k-1; W (k) is the weight matrix of layer k; b (k) is the bias vector of the kth layer.

[0121] b).Decoding process

[0122] The decoding process gradually transforms the feature layer Z into an approximation of the original input data through a symmetrical multi-layer structure. The formula is as follows:

[0123]

[0124] in:

[0125] l is the decoding layer index, which decreases from the feature layer to the output layer;

[0126] Outputs of the decoder layer l and layer l+1;

[0127] is the weight matrix of the decoder layer l;

[0128] is the bias vector of the decoder layer l;

[0129] f(·): Activation function, using ReLU function.

[0130] Final reconstructed output:

[0131]

[0132] c). Network optimization

[0133] The objective function is to minimize the reconstruction error And regularize and optimize the weights:

[0134]

[0135] in:

[0136] is the square of the two norm, representing the input data X and the reconstructed output The sum of squared Euclidean distances between

[0137] λ is the regularization coefficient, which is used to control the complexity of the model;

[0138] is the square of the Frobenius norm of the weight matrix of the lth layer.

[0139] The network extracts deep features of data layer by layer, which is suitable for processing structural and semantic differences between different data sources, and provides standardized feature representation for subsequent feature selection and mapping rule construction.

[0140] B2). Progressive feature selection

[0141] Progressive feature selection dynamically screens key features based on feature importance scores, effectively reducing redundancy and improving model efficiency.

[0142] a). Feature importance score

[0143] For the feature set F = {f1,f2,…,f m}, each feature f i Importance score S i The formula is as follows:

[0144] S i =αR i +βC i +γI i

[0145] in

[0146] R i: Relevance score, which measures the relevance of the feature to the target variable;

[0147] C i : Complexity score, reflecting the simplicity of eigenvalue distribution;

[0148] I i : Information gain, which measures the information contribution of features to the target variable;

[0149] α, β, γ: weight coefficients, satisfying α+β+γ=1.

[0150] b). Feature score calculation

[0151] Relevance score R i :

[0152]

[0153] Where: corr(f i ,Y) is the feature f i Pearson correlation coefficient with the target variable Y; m is the total number of features.

[0154] Complexity score C i :

[0155]

[0156] Where: H(f i ) is the feature f i The information entropy of ; N is the number of samples, that is, the total number of feature values.

[0157] Information Gain I i :

[0158] I i =H(Y)-H(Y|f i )

[0159] Where: H(Y): information entropy of target variable Y; H(Y|f i ): Given feature f i The conditional entropy of Y at .

[0160] c). Iterative selection strategy

[0161] In progressive feature selection, the feature selection process is carried out step by step through a forward search strategy, and each round of iteration selects a score S i The highest feature is added to the selected feature set F s , until the predetermined selection criteria or stop condition is met. The formula is as follows:

[0162]

[0163] in: is the feature selected in round t; F is the feature set, including all candidate features; F s is the selected feature set, which is dynamically updated in each iteration; S i is the importance score of the feature; argmax means selecting the feature with the highest score.

[0164] This mechanism automatically selects the optimal feature subset in dynamic adjustment, reduces dimensional redundancy, and improves feature quality and mapping efficiency.

[0165] C2). Mapping rule optimization

[0166] a). Mapping rule construction

[0167] In the intelligent data mapping engine, a mapping rule set R = {r1, r2, ..., r k}, each rule r i Both represent a mapping relationship from input data to output data. The optimization process of the rule set aims to maximize the overall performance of the mapping rules and adjust the rule weights w i To improve the accuracy and efficiency of mapping. Each rule is expressed as:

[0168] r i =(c i ,o i ,m i )

[0169] Where: c i is the rule condition vector, defining the mapping trigger condition; o i is the rule operation vector, defining the data standard after mapping; m i is the rule score, and the calculation formula is:

[0170]

[0171] Where n correct is the number of samples correctly mapped by the rule; n total is the total number of samples; |c i | is the rule condition number, indicating the complexity of the condition; |F s | is the total number of features, indicating the total number of selected features in the current model.

[0172] b). Rule set optimization

[0173] The rule score is optimized by linear programming, and the objective function is:

[0174]

[0175] Constraints:

[0176] in:

[0177] w i is the rule weight, which indicates the importance of the rule in the final decision; k is the number of rules in the rule set.

[0178] By optimizing mapping rules, the accuracy and stability of data standardization mapping can be improved to adapt to the diverse needs of complex scenarios.

[0179] 3. Elastic Resource Scheduler

[0180] The elastic resource scheduler is built on the YARN resource scheduling framework and adopts a microkernel architecture design. By integrating the biological cell division mechanism, entropy balance mechanism and resource prediction model, it realizes adaptive scheduling and dynamic expansion of computing resources. In a distributed environment, the scheduler optimizes the use of computing resources through multi-dimensional resource management and intelligent scheduling to ensure the balance and efficiency of resource supply. The specific implementation includes:

[0181] A3). Resource Management Model

[0182] In the process of computing resource management, the resource units in the distributed environment are abstracted into cell units with split characteristics. Each cell unit C is represented as a four-tuple:

[0183] C=(R,S,T,E)

[0184] in:

[0185] R: resource vector, describing the basic resource capacity of the cell unit, R = [r c ,r m ,r d ] T , where r c Indicates CPU resource capacity; r m Indicates memory resource capacity; r d Indicates storage resource capacity;

[0186] S: state vector, reflecting the resource usage of the cell unit, S = [s c ,s m ,s d ] T , where s c Indicates the real-time utilization of CPU resources; m Indicates the real-time usage of memory resources; d Indicates the real-time usage of storage resources;

[0187] T: Load threshold vector, defining the boundary conditions of resource usage, T = [t l ,t h ]T , where t l Indicates the load lower limit threshold; t h Indicates the upper load threshold;

[0188] E: Cell energy function, used to evaluate the operating status of resource units, the formula is:

[0189]

[0190] In the formula, α, β, γ are weight coefficients, and they satisfy the normalization condition α+β+γ=1; v is the resource utilization rate fluctuation value; v m is the fluctuation threshold; s i 、r i are the utilization rate and total capacity of the i-th resource unit, i∈{c,m,d}, corresponding to CPU, memory, and storage; max(·) represents the maximum value function.

[0191] B3). Mechanism of cell division

[0192] The cell division mechanism is responsible for managing the expansion and contraction of resource units. The judgment criteria are as follows:

[0193]

[0194] When the cell energy E exceeds the upper threshold t h When E is lower than the lower threshold t, the split operation is triggered, that is, the resource unit is divided into smaller units to expand the resources; when E is lower than the lower threshold t l When E is within the threshold range, the configuration of the current resource unit is maintained.

[0195] C3). Entropy balance mechanism

[0196] In order to ensure the overall balance of resources, an entropy evaluation mechanism is introduced to define resource entropy H r for:

[0197]

[0198] Among them, p i Indicates the resource proportion of the resource unit, obtained by weighted calculation:

[0199]

[0200] In the formula, w c ,w m ,w d is the weight coefficient of resource type; r c,i ,r m,i ,rd,i Respectively represent the CPU, memory and storage resource capacity of the i-th resource unit; r c,j ,r m,j ,r d,j They represent the CPU, memory and storage resource capacities of the jth resource unit respectively.

[0201] The system then evaluates the resource distribution status through resource balance B:

[0202]

[0203] in:

[0204] H max is the maximum entropy value, which indicates the entropy value when resources are completely evenly distributed. max =log2n, where n is the number of resource units. The optimization goal is to minimize the balance deviation. The formula is:

[0205] min|BB t |

[0206] in:

[0207] B t The target balance;

[0208] The constraints are:

[0209] in:

[0210] R total is the total resource capacity of the system; r min The minimum resource allocation unit is a single resource unit.

[0211] D3). Resource demand forecast

[0212] To predict future resource demand, the system uses LSTM (Long Short-Term Memory) neural network. LSTM is suitable for time series data, can capture long-term dependencies, and dynamically predict resource demand. The input of LSTM is historical resource utilization, load level, and historical resource allocation information:

[0213] x t =[u t ,l t ,h t ]

[0214] Where: u t represents the resource utilization rate at time step t; l t represents the load level at time step t; h t Represents the historical resource allocation information at time step t.

[0215] The output of LSTM is the predicted resource demand That is, the computational resources required at the future time step t. The training of the LSTM model is optimized by minimizing the mean square error (MSE) loss function:

[0216]

[0217] in:

[0218] y t For actual resource requirements; Forecasted resource requirements.

[0219] The Adam optimizer is used in the optimization process to accelerate convergence and improve training efficiency.

[0220] E3). Resource scheduling and dynamic expansion

[0221] In the resource scheduling process, the cell division mechanism is responsible for the expansion and contraction control of a single resource unit, the entropy balance mechanism ensures the balance of global resource distribution, and the resource prediction model LSTM provides a basis for predicting future resource demand. The three work together to dynamically adjust resource configuration, ensuring that the system can automatically expand under high load and release unnecessary resources under low load.

[0222] 4. Distributed storage center

[0223] The distributed storage center builds an efficient and reliable unified data storage platform based on HDFS (Hadoop Distributed File System) and data lake architecture. To further optimize storage efficiency and access performance, adaptive grid partitioning technology, deterministic chaos mapping and data temperature tiered storage mechanism are adopted. These technologies work together to achieve efficient data storage and concurrent access management.

[0224] Specific implementation includes:

[0225] A4). Adaptive mesh partitioning

[0226] In terms of storage optimization, adaptive grid partitioning technology achieves fine-grained control of data distribution by dividing the storage space into multiple dynamically adjusted grid units. The division of grid units is based on the distribution density function D(x,y) of the data block, which is formulated as follows:

[0227]

[0228] in:

[0229] (x i ,y i ) represents the storage location of the i-th data block; H iIt represents the access popularity of the data block and affects the storage density. δ(·) is a two-dimensional Dirac function, which represents the data block at the position. N is the total number of data blocks.

[0230] B4). Data block access heat calculation

[0231] The access popularity of the data block H i It is calculated by a weighted combination of three indicators: access frequency, relevance, and timeliness. The specific formula is:

[0232] H i =αf i +βr i +γt i

[0233] in:

[0234] f i is the access frequency of the data block, indicating the frequency of data block access; r i is the correlation degree of the data block, which is calculated by the cosine similarity between two data blocks, indicating the closeness of the relationship between the data block and other data blocks; i is the timeliness of the data block, reflecting the activity of the data block since the last access; α, β, γ are weight coefficients, and α+β+γ=1 is satisfied to ensure the balance of the three indicators.

[0235] By dynamically adjusting the size of the grid unit, the storage system can effectively respond to the density and activity of data access and optimize resource allocation. The adjustment of the grid unit is controlled by the following function:

[0236] G(x,y)=G min +(G max -G min )exp(-λD(x,y))

[0237] in:

[0238] G min and G max are the minimum and maximum sizes of the grid unit respectively; λ is the density response coefficient, which controls the degree of response of the grid unit size to the data distribution density.

[0239] C4). Deterministic chaotic mapping

[0240] In order to further optimize the distribution of data blocks, a two-dimensional Logistic mapping method is used. This chaotic mapping can ensure that the positions of data blocks are evenly distributed in the storage grid. The two-dimensional position update formula of the data block is as follows:

[0241] x n+1 =μ x x n (1-xn )

[0242] y n+1 =μ y y n (1-y n )

[0243] in:

[0244] x n and n is the location coordinate of the data block at the current moment;

[0245] μ x and μ y For the control parameters, the value range is [3.57,4] to ensure the chaotic characteristics.

[0246] The location mapping of data blocks is performed by the following function:

[0247]

[0248] in:

[0249] b i is the unique identifier of the data block; N x and N y The number of rows and columns of the storage grid respectively;

[0250] To round down, ensure that the data block is mapped to the appropriate grid cell; x n ,y n For data block b i The original mapping coordinates of .

[0251] D4). Phase space model of storage state

[0252] In order to fully describe the state of the storage system, a phase space model Φ of the storage state is constructed, which is defined as:

[0253] Φ={(x,y,v)|x∈[0,N x ],y∈[0,N y ],v∈V}

[0254] Where: (x, y) is the coordinate of the storage grid; v is the access speed of the data block, which means the access frequency of the data block in unit time; V is the vector space of the data block access speed.

[0255] In order to describe the dynamic evolution of the storage system, the following trajectory equation is used:

[0256]

[0257] Among them: A is the system state matrix, which represents the internal changes of the system state; B is the control matrix, which represents the influence of external control; u is the storage operation vector, which represents the system's control strategy for data storage.

[0258] E4). Concurrent access control and stability

[0259] In order to ensure that the storage system can operate stably under high concurrent access, the Lyapunov stability criterion is used for control. The Lyapunov function V(Φ) is defined as:

[0260] V(Φ)=Φ T PΦ

[0261] Its derivatives satisfy the following conditions:

[0262]

[0263] Where: P is a symmetric positive definite matrix, which is used to define the stability of the system; u is a storage operation vector, which represents the external control of the system behavior.

[0264] This stability criterion can effectively avoid resource contention and system crashes, ensuring stable operation of the system under high concurrency conditions.

[0265] F4). Data temperature tiered storage

[0266] The data temperature tiered storage mechanism uses the activity index A i Classify data blocks to ensure that different levels of data storage have different priorities. i The calculation formula is:

[0267]

[0268] in:

[0269] f i is the access frequency of the data block;

[0270] f max is the maximum access frequency;

[0271] t c is the current time, t i The last access time of the data block;

[0272] t max The maximum timeliness of all data blocks;

[0273] s i is the size of the data block;

[0274] s max is the maximum size of all data blocks;

[0275] ω1, ω2, ω3 are weight coefficients, and satisfy ω1+ω2+ω3=1, indicating the importance of each indicator.

[0276]

[0277] in:

[0278] θ h and θ l are temperature thresholds, representing the boundaries of the hot layer and the cold layer respectively.

[0279] G4). Data migration strategy between layers

[0280] The inter-layer data migration is controlled by the state transfer matrix P, which is in the following form:

[0281]

[0282] in:

[0283] p hh ,p hw ,p hc Represent the probability of data migration from the hot layer to the hot layer, warm layer, and cold layer respectively;

[0284] p wh ,p ww ,p wc Represent the probability of data migration from the warm layer to the hot layer, the warm layer, and the cold layer respectively;

[0285] p ch ,p cw ,p cc They represent the probability of data migration from the cold layer to the hot layer, warm layer, and cold layer respectively. ij

[0286] Through this migration strategy, the system can dynamically adjust the storage location of the data according to its activity level, thereby optimizing the utilization efficiency of storage resources.

[0287] 5. Business Analysis Engine

[0288] The business analysis engine is built on the SparkMLlib framework and integrates multi-scale feature fusion technology, graph projection technology, and knowledge reasoning engine to achieve intelligent analysis of enterprise business data. Through the organic collaboration of these three core technologies, the business analysis engine can extract multi-level features from data, build business status graphs, and support intelligent decision-making and analysis through knowledge reasoning. Specific implementation includes:

[0289] A5). Multi-scale feature fusion

[0290] Multi-scale feature fusion technology performs deep learning on input data through multi-level convolution and pooling operations to extract richer feature information and build a multi-layer feature extraction network, where the features output by each layer are represented as a feature mapping sequence F:

[0291] F={f1,f2,...,f n}

[0292] Among them, f i represents the feature map of the i-th layer, and n is the number of feature layers.

[0293] The feature map f of each layer i (x) is determined by the convolution operation and the pooling operation. The specific calculation formula is as follows:

[0294] f i (x) = σ(W i ·pool(conv(x,K i ))+b i )

[0295] Where: x is the input data; σ is the activation function, Sigmoid is selected; W i is the weight matrix of the i-th layer; b i is the bias term of the i-th layer; the convolution kernel matrix K i Perform convolution operation to obtain the convolution feature map to extract local features in the data, where K i is the convolution kernel matrix, defined as:

[0296]

[0297] In the formula, K i is a convolution kernel matrix of size m×m, each k ij It is the convolution kernel element, which determines the weight of each region during the convolution operation.

[0298] Pool(·) is a downsampling method, usually used to reduce the spatial dimension of data while retaining important feature information. Here, a dynamic weight pooling strategy is adopted, and the pooling formula is:

[0299]

[0300] In the formula, w ij is the pooling weight, which represents the weight of each element in the pooling process. The specific calculation is as follows:

[0301]

[0302] x ij 、x pqare the values ​​of the i-th, j-th elements and the p-th, q-th elements in the input data x; a is the sensitivity parameter of the pooling, which controls the sensitivity of the pooling to the feature. A larger a will make the pooling more sensitive to larger input values.

[0303] In order to extract more meaningful information from features at multiple levels, the attention mechanism is used for feature fusion. The fused features are represented as h i :

[0304]

[0305] In the formula, f j (x) is the feature map extracted from the jth layer, which is the feature representation obtained after the input data undergoes different convolution and pooling operations; n is the total number of feature layers, indicating the number of layers to be fused; α ij is the attention weight, which indicates the importance of the j-th layer feature map in the i-th layer and is calculated by the following formula:

[0306]

[0307] In the formula, e ij is the scoring function of feature fusion, and the calculation formula is as follows:

[0308] e ij =v T tanh(W[f i (x); f j (x)])

[0309] Where v is a learnable parameter vector; W is a weight matrix used to fuse features; [f i (x); f j (x)] means concatenating the feature maps of the i-th layer and the j-th layer as the input of the attention mechanism; tanh(·) is the hyperbolic tangent activation function, which is used to perform nonlinear transformation on the feature map.

[0310] Finally, through the attention mechanism, important features will receive higher weights, ensuring that more prominent features receive higher attention when features are fused.

[0311] B5). Atlas projection

[0312] Graph projection technology is used to model and analyze enterprise business data, represent and analyze entities in the enterprise and the relationships between them, and express business data in the form of a graph structure, including graph construction, graph embedding, and graph projection matrix construction, as follows:

[0313] a). Graph construction

[0314] The business graph G consists of three parts:

[0315] V: A collection of nodes in the graph, representing various entities in the business. Each node represents a business entity and is the basic unit of the graph.

[0316] E: The set of edges between nodes, representing the relationship between entities. Each edge represents some kind of interaction or relationship between two entities. Edges are the connection between nodes;

[0317] A: A set of attributes for each node and edge. The attributes of nodes and edges provide a detailed description of entities and relationships and are the basis for subsequent analysis.

[0318] b). Graph embedding target

[0319] The goal of graph embedding is to transform each entity node v in the graph i Mapping to a vector space allows the relationship between entities to be measured by the similarity of the vectors. The objective function in the graph embedding process is as follows:

[0320]

[0321] in:

[0322] Φ(v i ): node v i The embedding vector is the embedding vector of the node v i The result of low-dimensional representation indicates the position and relationship of the node in the graph structure;

[0323] N(v i ): with node v i The set of adjacent nodes, representing the nodes with v i Other nodes with direct relationships;

[0324] P(v j |Φ(v i )): conditional probability, representing node v j At a given node v i The embedding vector Φ(v i ), which reflects the probability of node v i and v j The similarity or correlation between them is calculated by the following formula:

[0325] In the formula, Φ(v j ) and Φ(v i ) are nodes v j and node v i Embedding vector of exp(Φ(v j ) T Φ(v i)) is used to calculate the embedding vector Φ(v i ) and Φ(v j ) is used to increase the weight of stronger relationships; k exp(Φ(v k ) T Φ(v i )) is a normalization term, which is used to ensure that the sum of all probabilities is 1 and avoid probability values ​​that are too large or too small. By optimizing this objective function, the graph embedding vector can be obtained so that similar nodes are closer in the embedding space and have a higher relationship probability.

[0326] c). Atlas projection matrix

[0327] The graph projection matrix P is constructed based on the structure of the business graph, and is mainly used to project the high-dimensional representation of the graph into a low-dimensional space for easy graph embedding and subsequent analysis. The graph projection matrix is ​​constructed using the following formula:

[0328]

[0329] Where: D is the degree matrix, which represents the degree of each node, that is, the number of edges directly connected to the node; A is the adjacency matrix, which represents the relationship between nodes in the graph. i With node v j If there is an edge connection, then A ij =1, otherwise 0; It is the inverse square root of the degree matrix D, which is used to standardize the adjacency matrix. Standardization can eliminate the influence of different node degrees, making the influence of each node more balanced in the projection. The business state vector s after graph embedding can be calculated by the following formula:

[0330] s=U T Φ(v)

[0331] Where U is the eigenvector matrix of the graph projection, obtained by spectral decomposition, which contains the low-dimensional representation of the graph structure; Φ(v) is the embedding vector of node v, indicating the position of the node in the low-dimensional space; U T Φ(v) represents mapping the embedding vector Φ(v) into a low-dimensional space, and the obtained s is the business state vector of the node.

[0332] C5).Knowledge Reasoning Engine

[0333] The knowledge reasoning engine realizes intelligent analysis and decision support of enterprise business knowledge by building ontology models and reasoning rules. Based on knowledge graph technology, the engine formalizes various business objects and their relationships of the enterprise, and mines implicit business rules and associations through reasoning mechanisms.

[0334] a). Ontology model and triple set

[0335] The ontology model is formally defined using a triple set O:

[0336] O={(e1,r,e2)|e1,e2∈E,r∈R}

[0337] in:

[0338] e1, e2: represent business entities, e1 is the head entity, e2 is the tail entity, representing different entity objects in the enterprise;

[0339] E: represents the business entity set, which includes all business entities and defines all business objects involved in the enterprise knowledge graph;

[0340] R: represents the set of all possible relationship types, defining all possible relationship types between entities in the graph.

[0341] b) Confidence evaluation of inference rules

[0342] In order to evaluate the effectiveness and reliability of the inference rules, the engine calculates the confidence conf(r) for each rule. The specific formula is:

[0343]

[0344] in:

[0345] Numerator: |{(e1,e2)|(e1,r,e2)∈O}| represents the number of entity pairs that satisfy rule r, that is, the number of head and tail entity pairs that meet the rule;

[0346] Denominator: represents the total number of head entities participating in rule r, that is, the denominator represents all possible head entities e1, which have at least one tail entity e2 associated with them.

[0347] The confidence conf(r) reflects the applicability of the rule in actual business scenarios. A rule with high confidence indicates that it is more common in the knowledge graph and applicable to more practical situations. Therefore, it will have a higher weight in the reasoning process.

[0348] c) Forward chain reasoning

[0349] The reasoning process uses the forward chain reasoning algorithm, and the matching probability P(r|h,t) of rule r is calculated as follows:

[0350]

[0351] in:

[0352] r ′represents all the rules in the rule set, which is used to compare and select the best rule; h represents the head entity, which is the current entity in the reasoning path; t represents the tail entity, which is the target entity in the reasoning path; f r (h, t): is the scoring function of rule r, which is used to measure the matching degree of rule r given the head entity h and the tail entity t. The rule scoring function is defined as:

[0353] f r (h,t)=r T [h; t]+b r

[0354] in:

[0355] r T represents the transpose of the weight vector of the rule, which is used to measure the importance of rule r in the current business scenario. Through model training and optimization, the weight vector can be automatically adjusted to adapt to actual business needs; [h; t] means concatenating the embedding vectors of the head entity h and the tail entity t to form a unified feature vector. This operation helps the system capture the relationship characteristics between entities; b r Represents the bias term of rule r, which is used to adjust the benchmark value of rule matching to ensure fairness for different rules during the reasoning process.

[0356] d). Reasoning path generation

[0357] The purpose of generating the reasoning path is to go from the head entity e1 to the tail entity e n A series of reasoning steps are performed to select a path with the highest comprehensive confidence. The calculation formula for path generation is:

[0358]

[0359] in:

[0360] e i ,e i+1 Represents the adjacent entity nodes in the path. The reasoning path starts from the head entity e1, passes through the intermediate entities in sequence, and finally reaches the tail entity e n ;

[0361] r i represents the rule used in the i-th reasoning step in the path;

[0362] P(r i |e i ,e i+1 ) represents the rule r in the i-th step of reasoning i The matching probability of

[0363] The matching probabilities of all reasoning steps are combined through multiplication operations, and finally the reasoning path p with the highest matching probability is selected, that is, the most likely business association in the path.

[0364] e). Decision support and filtering

[0365] Finally, the system filters the reasoning path according to the set confidence threshold θ and generates the final decision support result set:

[0366] D={d|conf(d)>θ,d∈path(e1,e n )}

[0367] in:

[0368] D represents the final decision support set, which contains all reasoning paths that pass the confidence filter; θ represents the confidence threshold, which ranges from [0,1] and is used to control the reliability of the reasoning results. Only when the confidence conf(d) of the reasoning path is higher than the threshold, the path will be included in the decision support set.

[0369] f). Practical application significance

[0370] The knowledge reasoning engine can provide intelligent decision support for enterprises by converting business knowledge into calculation rules. The following are several typical application scenarios:

[0371] Identify deep connections between products and optimize product mix strategies: By analyzing the purchase connections between products through the inference engine, companies can optimize product mix strategies, improve customer satisfaction and promote sales growth.

[0372] Mining customer behavior patterns and predicting potential business opportunities: The inference engine can predict customers’ potential needs based on their historical behavior patterns, helping companies identify potential business opportunities and conduct precision marketing.

[0373] Analyze business process bottlenecks and provide optimization suggestions: By reasoning and analyzing potential bottlenecks in business processes, enterprises can improve operational efficiency, reduce costs, and provide targeted optimization suggestions.

[0374] Early warning of potential business risks and support for proactive intervention: The business reasoning engine identifies potential risks, such as funding chain problems or supply chain disruptions, to help companies take timely intervention measures to avoid losses.

[0375] The knowledge reasoning engine provides intelligent support for business analysis and decision-making of enterprises through the formal representation of knowledge graphs, the construction and evaluation of reasoning rules. Through reasoning path generation and confidence filtering, the system can mine potential business rules from complex business data, support diversified business scenario applications, and help enterprises make better decisions and management.

[0376] Combination Figure 2 The workflow of the enterprise information management integrated platform described in the present invention is as follows:

[0377] S1. Data acquisition and parallel processing

[0378] Based on the parallel data processing center, the system first realizes the collection and processing of heterogeneous enterprise data; it builds a processing agent node network through swarm intelligence algorithm, and uses pheromone transmission mechanism to realize adaptive distribution of tasks to ensure load balancing of data processing tasks; for different types of data, it uses adaptive wavelet compression technology for unified processing: feature extraction and compression of structured business data, pattern recognition and normalization processing of semi-structured log data, and semantic analysis and structured conversion of unstructured document data; this stage establishes the basic framework for enterprise data processing.

[0379] S2. Data Mapping and Feature Extraction

[0380] After obtaining the preprocessed data, the intelligent data mapping engine realizes deep feature extraction through a hierarchical autoencoder network; the multi-layer encoder-decoder structure can automatically learn the intrinsic feature representation of the data and realize semantic-level mapping of heterogeneous system data; the progressive feature selection mechanism selects the most representative feature set through an iterative optimization process, and builds a unified data standard system for the enterprise based on it; this stage realizes the standardized expression and feature extraction of enterprise data.

[0381] S3. Resource Scheduling and Dynamic Expansion

[0382] The elastic resource scheduler continuously monitors the status of system resources and performs dynamic resource management based on the biological cell division mechanism. By abstracting computing resources into cell units, the system can automatically trigger resource splitting or merging operations according to the load conditions. The entropy balance mechanism ensures the balance of resource distribution and avoids resource hotspots. At the same time, the LSTM time series prediction model is used to predict resource demand trends, reserve and adjust resources in advance, and ensure the smooth operation and elastic expansion capabilities of the system.

[0383] 4. Distributed storage and management

[0384] The distributed storage center uses the HDFS file system and data lake architecture to build a unified storage platform; the adaptive grid partitioning technology dynamically adjusts the storage unit size according to the data access popularity, and the deterministic chaos mapping optimizes the data distribution strategy and improves the concurrent access performance; the system uses the data temperature stratification mechanism to store frequently accessed hot data in the high-performance storage layer, and migrates infrequently accessed cold data to the low-cost storage layer, thereby achieving efficient utilization and intelligent management of storage resources.

[0385] 5.Business analysis and decision support

[0386] The business analysis engine builds an analysis model based on the Spark MLlib framework, and realizes multi-level expression of data features through multi-scale feature fusion technology; it combines graph projection technology to project business entities and relationships into the feature space to build an enterprise operation monitoring model; and then uses the knowledge reasoning engine to conduct in-depth mining and association analysis of business data based on ontology models and reasoning rules, supporting multi-dimensional decision support including product association analysis, customer behavior prediction, process optimization suggestions, etc.

[0387] As shown in the above drawings, the enterprise information management integration platform based on big data provided by the present invention is built on the Hadoop ecosystem, and realizes the unified management and analysis of enterprise heterogeneous data through the collaborative work of five core modules: parallel data processing center, intelligent data mapping engine, elastic resource scheduler, distributed storage center and business analysis engine. The platform innovatively applies swarm intelligence algorithm and pheromone transmission mechanism to parallel data processing, adopts biological cell division mechanism to realize dynamic scheduling of resources, optimizes data storage through adaptive grid partitioning and deterministic chaos mapping, and integrates multi-scale feature fusion and knowledge reasoning technology to support intelligent decision-making. The platform effectively solves the problems of low data processing efficiency, poor system scalability, and difficulty in data sharing in traditional enterprise information management, and provides reliable big data technology support for enterprise digital transformation.

[0388] It is worth noting that the protection scope of the present invention is not limited to the specific implementations described in the above embodiments. Equivalent variations or modifications made by those skilled in the art based on the enlightenment of the present invention should be included in the protection scope of the present invention as long as they do not deviate from the spirit of the technical solution of the present invention.

Claims

1. An enterprise information management integration platform, characterized in that: It includes parallel data processing center, intelligent data mapping engine, elastic resource scheduler, distributed storage center and business analysis engine; among which: The parallel data processing center adopts the Hadoop distributed computing framework, sets up a swarm intelligence algorithm and a pheromone transmission mechanism to build a data processing agent node network, and combines adaptive wavelet compression technology to achieve unified processing of enterprise heterogeneous data; The intelligent data mapping engine adopts the Spark real-time computing framework, sets up a hierarchical autoencoding network with a multi-layer encoder-decoder structure, and combines a progressive feature selection mechanism to achieve adaptive mapping of enterprise heterogeneous system data; The elastic resource scheduler adopts the YARN scheduling framework, abstracts computing resources into cell units, triggers splitting or merging operations according to the cell energy function, and uses the entropy balance mechanism to achieve dynamic adjustment of resources; The distributed storage center adopts HDFS distributed file system, combines adaptive grid partitioning technology and deterministic chaos mapping to optimize data distribution, and realizes hierarchical storage of hot and cold data through data temperature stratification mechanism; The business analysis engine adopts the Spark MLlib framework, integrates multi-scale feature fusion technology and graph projection technology to build an enterprise operation monitoring model, and sets a knowledge reasoning engine to support business decision analysis.

2. The enterprise information management integrated platform according to claim 1, characterized in that: The swarm intelligence algorithm in the parallel data processing hub represents the data processing task as a five-tuple (id, P, R, D, S), including: Task unique identifier id; A priority vector P that describes task urgency, resource usage, and business importance; The resource vector R that describes the computing, storage, and network resource requirements; A set D describing the dependencies between tasks; Contains task identifiers S of pending, in-process, and completed states.

3. The enterprise information management integrated platform according to claim 1, characterized in that: The adaptive wavelet compression technology in the parallel data processing hub includes: Use discrete wavelet transform to perform multi-scale decomposition on the input data sequence; Dynamically calculate the compression threshold based on the standard deviation of the wavelet coefficients; Processing wavelet coefficients by soft thresholding method; The compression ratio is adjusted based on the information entropy ratio of the original data and the compressed data.

4. The enterprise information management integrated platform according to claim 1, characterized in that: The hierarchical self-encoding network in the intelligent data mapping engine includes: A multi-layer encoder structure with ReLU activation function is used for feature extraction; The multi-layer decoder structure symmetrical to the encoder realizes data reconstruction; The objective function combining reconstruction error and weight regularization is used for optimization.

5. The enterprise information management integrated platform according to claim 1, characterized in that: The cell unit in the elastic resource scheduler is defined as a quaternion C=(R, S, T, E), including: The resource vector R that describes the capacity of CPU, memory and storage resources; The state vector S that describes the real-time utilization of CPU, memory, and storage resources; The load threshold vector T that defines the CPU, memory, and storage scaling boundaries; Cell energy function E that comprehensively evaluates CPU, memory, and storage usage.

6. The enterprise information management integrated platform according to claim 1, characterized in that: The data optimization method in the distributed storage center includes: Construct grid cells based on data block distribution density function and access popularity; Use two-dimensional Logistic mapping to update the data block position; Implement temperature-tiered storage of data based on data activity indicators.

7. The enterprise information management integrated platform according to claim 1, characterized in that: The knowledge reasoning engine in the business analysis engine includes: The ontology model is constructed using triple sets; Screening inference rules through confidence evaluation mechanism; Generate reasoning paths based on the forward chaining algorithm.

Images