Big data intelligent association analysis decision-making method based on deep learning

Abstract

The invention relates to the technical field of big data analysis, in particular to a big data intelligent association analysis decision-making method based on deep learning, and the method comprises the steps: obtaining and aligning multi-source heterogeneous big data according to a preset rule, inputting an end-to-end training model, and then generating a high-order association feature vector representing an element association relationship through an association feature generation layer; information density indexes are obtained through internal state auditing; the decision output layer outputs a decision result based on the vector, and dynamically adjusts a weighting coefficient for balancing decision precision and feature efficiency in loss calculation according to an audit result; then combining a decision result, an adjustment coefficient and a true value to calculate a composite loss value, and finally synchronously updating two-layer model parameters; according to the method, dynamic collaborative optimization of rule mining and decision targets is realized; and a closed-loop mechanism is constructed through composite loss value feedback, weighting coefficient dynamic adjustment and parameter synchronous updating, so that a decision result reversely optimizes a correlation feature generation strategy.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, specifically to a deep learning-based intelligent correlation analysis and decision-making method for big data. Background Technology

[0002] Current big data intelligent decision-making systems generally adopt a phased processing architecture. First, independent association rule mining tools (such as the Apriori algorithm, FP-Growth algorithm, or their improved variants) extract high-frequency itemsets and strong association rules from multi-source data, generating a discrete symbolic rule base. Then, a rule engine filters some rules and transforms them into feature vectors, which are finally input into machine learning models (such as support vector machines, random forests, or neural networks) for decision prediction. This architecture has some technical shortcomings: The association rule mining process is completely independent of the decision-making task objectives, relying solely on statistical indicators such as support and confidence to screen rules. This results in a large number of rules that are irrelevant to the final decision being retained (for example, 80% of shopping association rules in financial risk control do not contribute to fraud identification), while key low-frequency, high-value association patterns are ignored, leading to a disconnect between objectives and contradictions. The training results of the decision model cannot be back-propagated to the rule generation stage. Once the rule base is generated, it becomes fixed and cannot be dynamically optimized according to the decision effect. This causes the rule base to be disconnected from the decision needs, resulting in a broken feedback chain. These shortcomings lead to bottlenecks in the system, such as the expansion of ineffective computation, limited decision-making accuracy, and poor scenario adaptability. There is an urgent need to break through the technical paradigm of separating rule mining and decision optimization. Summary of the Invention

[0003] The purpose of this invention is to provide a big data intelligent association analysis and decision-making method based on deep learning, in order to solve the problems mentioned in the background. Specific technical problems include how to achieve dynamic collaborative optimization between association rule generation and decision objectives, to solve the inefficiency and low accuracy problems caused by the disconnect between rule mining and decision requirements in traditional processes; and how to construct a closed-loop feedback mechanism to solve the fragmentation problem where the decision model cannot back-optimize the association pattern mining strategy.

[0004] To achieve the above objectives, the present invention provides a big data intelligent association analysis and decision-making method based on deep learning, comprising the following method steps: S1. Acquire multi-source heterogeneous big data, which includes structured data, time-series data, unstructured data, and image data; align and integrate different source data according to preset rules, which include one of timestamps, user identifiers, and device identifiers; align according to preset rules to eliminate data source differences and ensure input consistency.

[0005] S2. Input multi-source heterogeneous big data into a pre-built end-to-end training model. The end-to-end training model includes a sequentially connected correlation feature generation layer and a decision output layer. The correlation feature generation layer adopts a convolutional neural network structure, performing fixed-size convolutional kernel sliding calculations to capture local correlation features. The decision output layer adopts a fully connected network, configuring the output function according to the task type (classification or regression). This structural design ensures tight coupling between rule generation (correlation features) and decision output. For example, the convolutional neural network layer focuses on feature extraction to optimize decision accuracy, while the fully connected layer realizes the task-oriented transformation of decision. S3, the association feature generation layer processes data to generate high-order association feature vectors. These vectors directly represent the relationships between multiple elements in multi-source heterogeneous big data. Through internal state auditing and analysis of their distribution characteristics (such as information entropy, dimensionality variance, etc.), information density indices are obtained. This allows for real-time evaluation of the quality of association rules during the feature generation stage, preventing disconnects. For example, an excessively high information density index indicates rule dispersion, potentially affecting decision accuracy; an excessively low index indicates insufficient rules, leading to inefficient decision-making.

[0006] Internal state auditing is performed on high-order correlation feature vectors to analyze their distribution characteristics and obtain information density indices. The internal state audit calculates information density indices (such as dimensional mean, dimensional variance, information entropy, and feature proportion), and makes state decisions (such as insufficient feature representation or feature diffusion) based on a set of fixed threshold parameters. This directly supports dynamic collaborative optimization; for example, when the information entropy value is below the lower bound threshold, it is marked as insufficient feature representation, requiring stronger rule generation; when it is above the upper bound threshold, it is marked as feature diffusion, requiring simplified rules.

[0007] Further specify the adjustment process of the weighting coefficients (such as incrementing or decrementing operations), and ensure that the decision objective (such as high accuracy requirements) guides the rule mining strategy in reverse by adjusting the term used in the loss function to balance decision accuracy and feature efficiency (e.g., incrementing the weighting coefficients to improve feature efficiency when feature representation is insufficient).

[0008] S4. The internal state audit of the higher-order correlation feature vector provides real-time feedback signals (i.e., information density indicators), where the information density indicators are used to trigger state decisions (e.g., information entropy values ​​exceeding the upper bound indicate rule diffusion), which provides a basis for feedback; subsequently, when the decision output layer outputs the decision results, it directly adjusts the weighting coefficients (used for loss calculation) based on the internal state audit results, thereby mapping the decision performance (e.g., decreased accuracy) inversely to the feature efficiency problem (e.g., rule mining needs to be optimized). The adjustment process of the weighting coefficients is dynamically fed back based on state decisions (such as incrementing when a state of insufficient feature representation is activated), and feedback stability is ensured through preset boundary rules (setting upper and lower limits for change). For example, when an abnormal distribution state (low dimensionality and variance) is activated, the weight of the distribution constraint term in the feature efficiency loss term is increased, so that the back optimization of the decision model can specifically adjust the mining strategy of the associated feature generation layer. The adjustment of the weighting coefficients includes stability control operations (such as entering a freeze period when the number of consecutive adjustments in the same direction reaches a preset threshold, and locking the coefficient value), which prevents oscillations in the feedback process and ensures the reliability of the closed-loop mechanism; the freeze period design can avoid the fragmentation caused by over-optimization and maintain the continuity of feedback.

[0009] S5. The composite loss value combines the decision loss term (such as the cross-entropy loss function for classification tasks) with the feature efficiency loss term adjusted by weighted coefficients (including sparsity loss and redundancy loss). This makes rule generation and decision objectives mathematically coordinated, where the feature efficiency loss term optimizes rule quality (reduces redundancy), the decision loss term optimizes decision accuracy, and the adjustment of weighted coefficients ensures a dynamic balance between the two.

[0010] The composite loss value serves as a feedback carrier, combining the decision result (true value) with the feature efficiency state (such as the feature efficiency loss term including sparsity loss). The adjusted weighting coefficient is directly incorporated into the calculation, so that the decision error is fed back to the rule mining layer (for example, when the redundancy loss is too high, the feedback association pattern needs to be simplified).

[0011] S6. The gradient components of the composite loss value with respect to the parameters of the associated feature generation layer and the decision output layer are calculated through the backpropagation algorithm, and the parameters are updated synchronously using the adaptive momentum optimization rule (covering the convolution kernel weights of the associated feature generation layer, the parameters of the normalized layer, and the weights and bias vectors of the decision output layer). This realizes the final step of closed-loop feedback. The decision loss is backpropagated through the composite loss value to update the entire model, so that the parameters of the associated feature generation layer (such as the convolution kernel) can be re-optimized according to the decision requirements to optimize the mining strategy (e.g., improve the feature extraction of convolution operations).

[0012] The above technical solution addresses the problems of inefficiency and low accuracy caused by the independence of association rule mining from the decision-making module in traditional methods, which prevents rule generation from responding to decision-making needs in real time. It also solves the problem of the decision model's inability to provide feedback and optimize the association mining process, leading to a fragmented approach. By providing feature status feedback through internal status auditing and adjusting weighted coefficients, the decision problem is transformed into a feature optimization signal. A composite loss value and a synchronous update mechanism form a closed-loop cycle from decision output feedback to association feature generation (e.g., low decision accuracy triggers coefficient adjustment, re-adjusting the feature efficiency loss term), ultimately achieving dynamic self-optimization of the association pattern mining strategy. This closed-loop mechanism avoids the fragmentation of traditional methods and improves the overall model's efficiency and accuracy.

[0013] Compared with the prior art, the present invention has the following beneficial effects: By leveraging the convolutional operations of the associated feature generation layer, high-order associated feature vectors are automatically generated. The quality of rules is evaluated in real time based on internal state review. Combined with a dynamic adjustment mechanism of weighted coefficients, the decision accuracy and feature efficiency are balanced, enabling the association rule mining to directly respond to decision needs, eliminating the disconnect between rule generation and decision objectives in the traditional process, and improving analysis efficiency and decision accuracy. By associating the decision results with the rule quality through a composite loss value, and using a synchronous update mechanism to backpropagate the gradient to the associated feature generation layer and the decision output layer, a closed loop of "decision feedback → rule reconstruction" is formed, breaking down the barrier between the decision model and the associated pattern mining. The freeze period control of weighted coefficients and boundary rule constraints ensure the stability and reliability of the feedback process, while the multi-source data alignment and integration and residual learning framework guarantee the robustness of feature generation and gradient backpropagation, ultimately achieving adaptive co-evolution of association rules and decision objectives. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall method steps of the present invention; Figure 2 This is the core flowchart of the present invention; Figure 3 This diagram illustrates the synergistic optimization effect of decision accuracy and feature efficiency in this invention. Detailed Implementation

[0015] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Next, please refer to Figure 1One of the objectives of this embodiment is to provide a big data intelligent association analysis and decision-making method based on deep learning, which includes the following steps: S1. In the data acquisition and preprocessing stage, heterogeneous data types are collected from multiple sources, and structured data is extracted from relational databases, including transaction record tables and sensor value tables with clearly defined fields; time series data streams are accessed through the data bus, typically represented by stock price fluctuation data streams and network traffic time series records; text type data is read from the unstructured storage pool, covering user comment texts and system log files; and visual image data in the monitoring scenario is obtained by calling the image storage service interface.

[0017] To eliminate analytical biases caused by data heterogeneity, a strict unified alignment operation is performed. For source data containing time information (such as sensor time-series records and transaction records), a global time-series anchoring mechanism is adopted, that is, time axis calibration with millisecond-level precision is performed based on a unified timestamp field. For data involving user entities (such as comment text and transaction records), identity consistency association is achieved through a user identifier mapping engine to establish a user-level data topology. For device-generated data (such as sensor data and device monitoring images), cross-source data binding is completed based on a device identifier hash matching algorithm.

[0018] The alignment process generates a dimensionally standardized sample matrix, in which structured numerical values ​​are compressed to a preset range through linear transformation, text data is encoded into word frequency vectors using a fixed vocabulary, and image data is downsampled into uniform resolution grayscale images. Finally, a standard sample set with four-dimensional feature channels is generated as multi-source heterogeneous big data. The accurate cross-source data association capability established by this process lays a structurally consistent foundation for subsequent deep association mining and solves the problem of missing key associations caused by data silos in traditional analysis methods.

[0019] S2. Construct and initialize an end-to-end deep learning training model, which consists of a sequentially connected associated feature generation layer and a decision output layer, wherein: The associated feature generation layer is designed as a convolutional neural network structure. The first convolutional module uses a preset number (e.g., 64 channels) of fixed-size convolutional kernels (e.g., 3×3) and performs sliding calculations with a preset stride (e.g., single-pixel step) to extract spatial association patterns from the input data. After each convolutional operation, a normalization layer is embedded to perform feature distribution calibration (batch normalization algorithm) and a non-linear activation function unit (linear rectified activation function) is connected. The second convolutional module expands the number of channels to a preset value (e.g., 128 channels) to enhance the receptive field range by expanding the convolutional operation. The final layer adds a spatial attention mechanism module. The convolutional output is compressed into a fixed-dimensional feature vector (e.g., 256 dimensions) by global average pooling. The decision output layer is constructed as a fully connected network. The number of hidden neurons in the first layer is configured according to a preset ratio (such as a multiple of the output dimension of the convolutional layer) and a self-regularized activation function is used. The number of neurons in the second layer is dynamically adjusted according to the task type (the number of output neurons for classification tasks equals the number of categories, and the output of a single neuron for regression tasks). The output layer deploys a classification function (flexible maximum probability transformation) or a regression unit (linear output module), the selection of which is predefined by the system task configuration file. During the initialization phase, the weights of the convolutional kernels are initialized using a normal distribution initialization algorithm with specific variance control, and the bias vector is set to zero. The weights of the fully connected layer are initialized using a uniform distribution initialization method. Through the synergy of the local correlation capture capability of the convolutional layer and the global decision-making capability of the fully connected layer, a complete mapping channel from raw data to decision results is formed.

[0020] The model connection method is that the output tensor of the associated feature generation layer is directly transmitted to the input of the decision output layer. The weight initialization follows a standardization process. The convolution kernel weights are initialized using a truncated normal distribution, and the mean and variance control parameters are set. The bias vector is initialized to zero. The weights of the fully connected layer are generated using a normal distribution with specific variance constraints. The dimension of the input layer is strictly consistent with the dimension of the preprocessed sample data, forming an end-to-end data flow channel.

[0021] S3. Process multi-source heterogeneous big data through the association feature generation layer to generate high-order association feature vectors representing the association relationships between multiple elements in the multi-source heterogeneous big data, specifically including: First, multi-level convolution operations are performed on multi-source heterogeneous big data. Each layer uses a convolution kernel with a preset stride and padding rules to extract local features. Each convolutional layer is followed by a normalization layer to calibrate the feature distribution. Element-level transformations are performed using a non-linear activation function. The final convolutional output is reduced to an initial feature vector of a predetermined length through global spatial compression. The feature enhancement stage employs a residual learning framework, where the initial feature vector is input into a feedforward network for nonlinear transformation. This network consists of two fully connected layers, with the intermediate layer using a standard activation function and the output layer applying a linear transformation. The network output is element-wise summed with the initial vector to form residual connections. Finally, a high-order correlation feature vector with a fixed dimension is generated, whose numerical range is constrained by the characteristics of the activation function. Each dimension represents the encoding of the nonlinear relationship between data elements. The entire process is a deterministic computation graph without random operations.

[0022] Internal state auditing is performed on high-order correlation feature vectors to analyze their distribution characteristics and obtain information density indices, specifically including: Monitor the state and quality of the high-order related feature vectors generated by the related feature generation layer during the training process to avoid them falling into low information content, high redundancy or collapse state (such as features being too concentrated or sparse), and provide a basis for subsequent adjustment and optimization direction. During model training, its distribution characteristics are analyzed to obtain information density indicators. Four core operations are performed on each batch of high-order correlation feature vectors to obtain information density indicators, including dimensional mean, dimensional variance, information entropy, and feature proportion. The dimensional mean is calculated by averaging all elements using the arithmetic mean formula; the dimensional variance is calculated based on the standard deviation squared formula; the information entropy is calculated by first normalizing the absolute value of the vector to form a probability distribution, then applying the Shannon entropy formula and dividing by the maximum entropy value for normalization; the feature proportion is calculated by setting an absolute value judgment threshold and counting the proportion of dimensions exceeding the threshold. The internal status review sets a fixed threshold parameter group, defines the lower and upper bound thresholds of the effective range of information entropy values, sets a minimum allowable value for variance, and specifies the qualified standard for feature ratio. The status judgment logic is as follows: when the information entropy value is lower than the lower bound threshold, it is marked as a state of insufficient feature expression; when the information entropy value is higher than the upper bound threshold, it is marked as a state of feature diffusion; when the dimensionality variance is lower than the minimum allowable value, it is marked as a state of abnormal distribution; when the feature ratio does not meet the standard, it is marked as a state of insufficient significance. The output adopts multi-bit combination encoding, and each status bit corresponds to a preset judgment condition.

[0023] S4. The decision output layer receives high-order correlation feature vectors and outputs decision results corresponding to the target decision task, specifically including: In the decision output layer, a deterministic mapping operation is performed. The higher-order correlation feature vector is first input into the first fully connected layer, where weight matrix multiplication and bias vector addition are performed to generate an intermediate feature vector. Element-wise transformation is then performed using a standard nonlinear activation function. The intermediate feature vector is then input into the second fully connected layer to perform dimensionality transformation and output the original prediction vector. The process involves task adaptation, where classification tasks apply a flexible maximum function to transform the original prediction vector into a category probability distribution, with the sum of probabilities remaining constant; regression tasks retain the original prediction values ​​as continuous outputs; specific application scenarios are configured with dedicated output structures, such as risk level prediction outputting a multi-level probability distribution, and equipment status detection outputting binary discriminant values; all parameters are trainable variables, the forward propagation process does not include random operations, the output dimension is predefined by the task configuration file, and the final output is the decision result corresponding to the target decision task.

[0024] Based on the internal state review results of the higher-order correlation feature vectors, the weighting coefficients used to balance decision accuracy and feature efficiency in the loss calculation process are adjusted, specifically including: The loss weighting coefficient is set to a preset baseline value, and adjustments are made based on the state review results. When the state of insufficient feature expression is activated, an incremental operation is performed according to the preset step size coefficient; when the state of feature diffusion is activated, a decrement operation is performed according to the same step size; the state of abnormal distribution triggers the enhancement of feature distribution constraint terms; the state of insufficient significance activates the sparsity penalty weight. The weighted coefficient adjustment follows preset boundary rules, setting upper and lower limits for weighted coefficient changes to ensure that the changes are within the effective range; the update process includes stability control, when the number of consecutive adjustments in the same direction reaches a preset threshold, a freeze period is entered, during which the coefficient value is locked; after the freeze is lifted, normal adjustment logic is restored; the adjustment direction and magnitude are uniquely determined by the combination of status code bits, and each adjustment decision is a deterministic operation under preset rules; The adjustment of the weighting coefficients enables the loss function to dynamically adapt to the current state of the associated feature generation layer. It establishes a direct feedback link from the state of the associated features to the fine-tuning of the optimization target, ensuring that the training process can automatically correct itself based on the quality of the high-order associated feature vectors, and guiding the associated feature generation layer to produce feature representations that are both relevant and efficient for the final decision task.

[0025] S5. Based on the decision outcome, the adjusted weighting coefficients, and the true value of the target decision task, calculate the composite loss value, specifically including: In the composite loss calculation stage, a multi-objective loss function is constructed. This function comprises two main parts: a decision loss term and a feature efficiency loss term. The decision loss term is configured according to the differences in task type. For classification tasks, the cross-entropy loss function is used, and the discrimination error is measured by calculating the log difference between the predicted probability distribution and the true label distribution. The true label is input in one-hot encoding format, and the predicted probability is generated by the flexible maximum function. For regression tasks, the piecewise smooth loss function is applied. When the absolute value of the predicted residual is less than a set transition threshold, the squared error is used for calculation. When the residual exceeds the threshold, the absolute value error is used for calculation, which enhances the model's ability to adapt to outliers. The feature efficiency loss term consists of two parts: the sparsity loss calculates the sum of the absolute values ​​of each dimension of the higher-order correlated feature vector, which promotes the simplification of feature representation; the redundancy loss calculates the absolute average value of the correlation between different dimensions of the feature vector, which suppresses information duplication. The final composite loss value is subjected to a linear combination operation, which multiplies the sum of the decision loss term and the feature efficiency loss term by an adjusted weighting coefficient. The sparsity loss and redundancy loss in the feature efficiency loss term are mixed according to a preset weight ratio. The gradient signal generated by this loss structure carries the dual adjustment effect of target discrimination requirements and feature quality optimization instructions during backpropagation, forming a synergistic optimization mechanism for accuracy and efficiency.

[0026] S6. Using the composite loss value, synchronously update the model parameters of the associated feature generation layer and the decision output layer, specifically including: During the model parameter synchronization update phase, the backpropagation algorithm is executed first: starting with the composite loss value, the partial derivatives of the loss function with respect to each parameter of the associated feature generation layer are calculated layer by layer using the chain rule, including the gradient components of each element in the convolution kernel weight matrix and the gradients of the scaling and offset parameters within the normalized layer; the gradient components of the loss function with respect to the fully connected weight matrix and bias vector of the decision output layer are calculated simultaneously; the gradient calculation process covers the connected paths of the entire computation graph to ensure that the gradient signals of the associated feature generation layer and the decision output layer are from the same source; Subsequently, an adaptive momentum optimization algorithm is used to update the parameters, maintaining the gradient first-order moment estimate and second-order moment estimate vectors for each parameter, and performing an exponential moving average based on a preset decay coefficient; bias correction is applied to the moment estimates to eliminate the influence of initial zero values; after calculating the adaptive learning step size, all weight parameters are updated. The update operation simultaneously applies to the trainable parameters (convolutional kernel weights, normalized layer parameters) of the associated feature generation layer and the trainable parameters (fully connected layer weights and bias vectors) of the decision output layer; this mechanism ensures that in each iteration, the decision optimization signal carried by the composite loss value and the feature efficiency constraint simultaneously react on the feature generation strategy and the decision mapping rule, achieving the co-evolution of the parameters of the two layers.

[0027] Please see Figure 2 The complete workflow of an end-to-end machine learning model, from data input to parameter updates, is described in detail below: It starts with S1 to acquire multi-source heterogeneous big data (integrating diverse data sources), and then enters S2 to input end-to-end training model (initializing the model training framework). The core components include the S3 associated feature generation layer (dynamically generating high-order associated feature vectors), which incorporates an internal state review mechanism—making a state decision (identifying feature validity) by calculating information density indicators (such as feature entropy and distribution dispersion); the decision result triggers S4 to dynamically adjust the weighting coefficients and optimize the loss function weights in real time; the decision layer (S4 decision output layer) outputs decision results (such as classification prediction) based on the weighted features. Subsequently, the S5 composite loss calculation (integrating decision loss and feature constraint loss) is performed. Finally, in the S6 synchronous parameter update stage, the parameters of the decision output layer (fully connected weights) and the parameters of the associated feature generation layer (feature extraction weights) are updated synchronously to form a closed-loop optimization.

[0028] This closed-loop training mechanism achieves the following core effects through dynamic feedback and dual-objective collaborative optimization: In complex data environments, it relies on an internal state audit module to evaluate the quality of high-order correlation features in real time (e.g., identifying insufficient or redundant representations through feature entropy and distribution dispersion), and simultaneously triggers a dynamic adjustment mechanism for weighting coefficients (reducing weights when features are diffuse and strengthening constraints when distributions are abnormal), enabling the composite loss function (decision loss + feature constraint loss) to simultaneously optimize decision accuracy and feature efficiency; furthermore, through parameter collaborative updates (synchronously updating the convolutional kernels of the feature generation layer and the fully connected weights of the decision layer in reverse), it ensures that the feature generation strategy and the decision objective are dynamically aligned, wherein: When feature redundancy leads to decision-making errors, the feature efficiency loss weight is automatically reduced to compress redundant features, thereby enabling the decision-making objective to guide rule mining in real time and solving the problem of inefficiency and low accuracy caused by the disconnect between rules and requirements in traditional processes. When decision errors originate from rule quality defects (such as invalid association features generated by S3), the gradient calculation of S6 directly corrects the parameters of the feature generation layer (e.g., by adding convolution kernel sparsity constraints), enabling the backpropagation of the decision model to feed back into the rule mining strategy in real time, thus breaking the fragmented dilemma in traditional schemes where the decision model cannot optimize the rule generation layer.

[0029] Ultimately, the closed-loop self-optimizing circuit significantly improves the model's generalization ability and feature interpretability, solving the problems of low expression efficiency and overfitting risk caused by the separation of feature engineering and model training in traditional methods.

[0030] Please see Figure 3 By observing the synergistic changes of two key performance indicator curves (decision accuracy and feature efficiency) throughout the training process, the working mechanism of the closed-loop optimization mechanism is revealed. In the initial stage (rounds 1-25), feature efficiency rapidly improves, verifying the effect of the weighting coefficient automatically strengthening feature constraints when the internal state audit detects insufficient feature representation. In the middle stage (rounds 25-60), decision accuracy significantly increases, reflecting that after the system recognizes that feature optimization has reached the target, it automatically shifts the optimization focus to decision accuracy. In the later stage (after round 60), both indicators converge to a high-level equilibrium region, proving that the method ultimately achieves Pareto optimality for both accuracy and efficiency, breaking through the technical bottleneck of traditional methods that struggle to balance both.

[0031] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A big data intelligent association analysis and decision-making method based on deep learning, characterized in that, The methods and steps include the following: S1. Acquire multi-source heterogeneous big data; S2. Input the multi-source heterogeneous big data into a pre-constructed end-to-end training model, wherein the end-to-end training model includes a sequentially connected associated feature generation layer and a decision output layer; S3. Process multi-source heterogeneous big data through the aforementioned correlation feature generation layer to generate a high-order correlation feature vector that represents the correlation relationship between multiple elements in the multi-source heterogeneous big data; perform internal status review on the high-order correlation feature vector and analyze its distribution characteristics to obtain information density index. S4. Receive the higher-order correlation feature vector through the decision output layer and output the decision result corresponding to the target decision task; adjust the weighting coefficient used to balance decision accuracy and feature efficiency in the loss calculation process according to the internal state review result of the higher-order correlation feature vector. S5. Calculate the composite loss value based on the decision result, the adjusted weighting coefficient, and the true value of the target decision task; S6. Using the composite loss value, synchronously update the model parameters of the associated feature generation layer and the decision output layer.

2. The big data intelligent association analysis and decision-making method based on deep learning according to claim 1, characterized in that, The multi-source heterogeneous big data includes structured data, time-series data, unstructured data, and image data; different source data are aligned and integrated according to preset rules, which include one of timestamps, user identifiers, and device identifiers.

3. The big data intelligent association analysis and decision-making method based on deep learning according to claim 1, characterized in that, The associated feature generation layer adopts a convolutional neural network structure, which includes multiple convolution operation modules. Each layer is configured with a preset number of convolution channels and performs sliding calculations of convolution kernels of fixed size. Normalization layers and nonlinear activation function units are embedded between layers. The decision output layer adopts a fully connected network structure. The number of neurons in the hidden layer is configured according to a preset ratio. The output layer selects either a classification function or a regression unit according to the task type.

4. The big data intelligent association analysis and decision-making method based on deep learning according to claim 1, characterized in that, The process of generating the higher-order correlation feature vector specifically includes: Multi-level convolutional operations are performed, with each layer using a convolutional kernel with a preset stride and padding rules to extract local features. The output of the final convolution is reduced to an initial feature vector of a predetermined length through global spatial compression. A residual learning framework is used to perform a nonlinear transformation on the initial feature vector to form residual connections, ultimately generating a high-order correlated feature vector of fixed dimensions.

5. The big data intelligent association analysis and decision-making method based on deep learning according to claim 1, characterized in that, The internal state audit includes calculating information density indicators and setting a fixed threshold parameter set. The information density indicators include dimensional mean, dimensional variance, information entropy value, and feature ratio. The setting of the fixed threshold parameter set is used for state judgment, specifically including: When the information entropy value is below the lower bound threshold, it is marked as a state of insufficient feature expression; when the information entropy value is above the upper bound threshold, it is marked as a state of feature diffusion; when the dimensionality variance is below the minimum allowable value, it is marked as a state of abnormal distribution; when the feature ratio does not meet the standard, it is marked as a state of insufficient significance.

6. The big data intelligent association analysis and decision-making method based on deep learning according to claim 1, characterized in that, The adjustment process for the weighting coefficients specifically includes: When the feature representation is insufficient, an increment operation is performed according to the preset step size coefficient; when the feature diffusion is active, a decrement operation is performed according to the preset step size coefficient; when the distribution is abnormal, the weight of the feature distribution constraint term in the feature efficiency loss term is increased; when the significance is insufficient, the sparsity penalty weight in the feature efficiency loss term is increased.

7. The big data intelligent association analysis and decision-making method based on deep learning according to claim 6, characterized in that, The adjustment of the weighting coefficients includes a stability control operation. When the number of consecutive adjustments in the same direction reaches a preset threshold, a freeze period is entered, during which the coefficient values ​​are locked.

8. The big data intelligent association analysis and decision-making method based on deep learning according to claim 6, characterized in that, The adjustment of the weighting coefficients follows preset boundary rules, including setting upper and lower limits for the changes.

9. The big data intelligent association analysis and decision-making method based on deep learning according to claim 1, characterized in that, The composite loss value is a linear combination of the decision loss term and the feature efficiency loss term adjusted by the weighted coefficients; the decision loss term adopts the cross-entropy loss function in the classification task and the piecewise smooth loss function in the regression task; the feature efficiency loss term includes sparsity loss and redundancy loss.

10. The big data intelligent association analysis and decision-making method based on deep learning according to claim 1, characterized in that, The synchronization update process specifically includes: The backpropagation algorithm is executed to calculate the gradient components of the composite loss value with respect to the parameters of the associated feature generation layer and the decision output layer. The parameters are updated using an adaptive momentum optimization rule, and the update range covers the convolution kernel weights of the associated feature generation layer, the normalized layer parameters, and the weights and bias vectors of the fully connected layer of the decision output layer.

Images