Analysis method for applying dense model to big data

Abstract

The invention discloses an analysis method for applying a dense model to big data, and the method comprises the following steps: S1, collecting and preprocessing heterogeneous original data, and generating a standardized data set; s2, performing feature extraction and feature splicing on different types of fields in the standardized data set to form a preliminary feature vector matrix; s3, inputting the initial feature vector matrix into a heterogeneous feature fusion and completion algorithm to generate a fused feature vector matrix; s4, inputting the fusion feature vector matrix into a deep dense residual network model, and outputting deep semantic feature representation; s5, inputting the deep semantic feature representation into a full-connection output layer to generate an analysis result vector; s6, constructing a loss function, and executing back propagation to optimize weight parameters in the deep dense residual network model; and S7, repeatedly executing the steps S1 to S6 on the newly added data. According to the method, deep residual network modeling and a swarm intelligence anomaly recognition mechanism are fused, and unified modeling, deletion completion and high-dimensional feature expression analysis of big data multi-source fields are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and big data analysis technology, and in particular to an analysis method for applying a dense model to big data. Background Art

[0002] With the continuous acceleration of big data technology, artificial intelligence algorithms, and the digital transformation of industries, enterprises, governments, and research institutions have accumulated massive amounts of heterogeneous data in various business and research scenarios. This data is widely distributed across multiple subsystems, including financial systems, sensor terminals, network platforms, and operational platforms, and its data types include complex modalities such as structured tables, semi-structured logs, unstructured text, and images. To extract high-value information from this heterogeneous data and form actionable analytical decisions, various big data analysis models have developed rapidly. In particular, data modeling methods based on deep neural networks have been widely used in tasks such as public opinion monitoring, risk warning, intelligent recommendation, and behavior prediction. However, although existing big data analysis systems have achieved data-driven intelligent modeling to a certain extent, they still face a series of technical bottlenecks that need to be addressed when facing practical problems such as data heterogeneity, multi-source missing, semantic inconsistency, and high dimensional coupling.

[0003] The current mainstream data analysis process generally uses manual or rule-based data cleaning, field mapping, and feature engineering methods to perform preliminary processing on heterogeneous data, and then inputs the preprocessing results into traditional neural network models (such as multi-layer perceptrons, convolutional neural networks, or long short-term memory networks) to perform analysis tasks. This method can achieve basic modeling in a clearly structured data environment, but in actual environments such as the integration of multiple business systems, the mixing of multi-format fields, and the interference of multiple missing dimensions, it is easy to cause problems such as loss of relationships between features, failure to align key fields, or insufficient accuracy of analysis results. For example, synonymous fields between different systems may not be correctly aligned due to naming differences, resulting in the data representation learned by the model being unable to maintain semantic consistency; some systems have a large number of missing fields or non-standard formats, making it difficult to directly use the original data for training; in addition, there are statistical distribution differences between data from different sources. If they are directly spliced ​​into the input model, it is easy to cause feature offset, affecting the stability of prediction.

[0004] To address the above problems, researchers have attempted to introduce feature alignment, data completion, and multimodal fusion mechanisms into deep learning frameworks in recent years. For example, these methods employ autoencoders to reconstruct missing values, graph neural networks to process multi-relational fields, or attention mechanisms to weight data channels from different sources. However, these methods are often limited to local improvements and lack a complete and unified analysis framework. At the same time, many models rely on massive amounts of complete data for end-to-end training. Once the input data contains many gaps or semantic conflicts, the training difficulty increases significantly and the generalization ability decreases. In addition, most methods process input features in a relatively static manner, making it difficult to perform dynamic modeling of key fields based on the business characteristics of different scenarios, resulting in a lack of adaptability in the analysis strategy.

[0005] On the other hand, most of the current mainstream deep models use shallow connection structures or unidirectional information pathways, lacking cross-layer feature aggregation and feature reuse mechanisms, which is not conducive to improving the model's expressive power and information fusion efficiency. In the feature extraction stage, convolution operations often find it difficult to fully model the cross-dimensional semantic relationships between different fields, and tend to ignore the ability to perceive long dependency structures. Although traditional residual connections can alleviate the problem of vanishing gradients, in the scenario of heterogeneous field fusion, there is still a lack of a mechanism for semantic-level completion and alignment of input features. In addition, the splicing fusion or weighted averaging methods commonly used in existing methods are also difficult to fully retain the semantic differences between information from different sources, which can easily lead to the dilution of important features and a decrease in the model's learning ability.

[0006] In general, existing big data analysis methods have the following problems when dealing with complex, heterogeneous and missing mixed data environments: First, there is a lack of a unified data alignment and completion mechanism, which cannot effectively fuse multi-source heterogeneous data, resulting in unstable input quality; second, there is a lack of dynamic feature modeling capabilities, and the analysis strategy cannot be automatically adjusted according to the input data content; third, the information transmission path in the deep model structure is single, and it is difficult to establish cross-level and cross-modal feature dependency expressions; fourth, the feature fusion method is simple and fails to fully explore the semantic structure and interaction relationship between multimodal features.

[0007] Therefore, there is an urgent need to design a unified modeling framework that integrates feature extraction, missing completion, dimension alignment and fusion enhancement, which can fully explore the potential correlation relationships in heterogeneous data, automatically adapt to multi-scenario analysis tasks, and improve the accuracy and robustness of data analysis. In response to the above-mentioned deficiencies in the existing technology, the present invention proposes a big data analysis method based on a deep dense residual network and a heterogeneous feature fusion completion algorithm. Through a modular data processing structure and a dense connection mechanism, it effectively solves the information integration and feature modeling problems of existing models in heterogeneous input environments, and has higher data adaptability and generalization capabilities. Summary of the Invention

[0008] One purpose of the present invention is to propose a big data analysis method based on a deep dense residual network and a heterogeneous feature fusion and completion algorithm. The present invention constructs a unified feature vector generation mechanism, generates a semantically consistent fusion feature vector matrix through standardization processing, multi-type field encoding, feature alignment and missing interpolation and completion operations, and introduces multi-layer dense connections and cross-layer residual transfer structures to extract deep semantic features. At the same time, it combines full connection mapping and supervised loss optimization to realize analysis result output and parameter update. It has the advantages of adapting to multi-source heterogeneous data, enhancing feature semantic consistency, supporting high-dimensional feature expression and scalable deployment, and is suitable for big data intelligent analysis scenarios such as financial risk control, operation optimization, and behavior recognition.

[0009] A method for analyzing big data using a dense model according to an embodiment of the present invention includes the following steps:

[0010] S1, collect heterogeneous raw data, perform preprocessing, and generate standardized data sets;

[0011] S2. Extract features from different types of fields in the standardized data set to obtain dense representation vectors, and perform feature concatenation to form a preliminary feature vector matrix;

[0012] S3. Input the preliminary feature vector matrix into the heterogeneous feature fusion and completion algorithm, and perform feature dimension alignment, missing feature interpolation completion, and semantic homogeneous feature mapping operations in sequence to generate a fused feature vector matrix;

[0013] S4. Inputting the fused feature vector matrix into a deep dense residual network model, wherein the deep dense residual network model is composed of multiple groups of dense connection modules and residual connection modules. The dense connection modules perform continuous feature splicing, convolution transformation and nonlinear activation operations, and the residual connection modules perform cross-layer feature superposition and output deep semantic feature representation;

[0014] S5. Input the deep semantic feature representation into the fully connected output layer, perform matrix multiplication and bias addition operations, and generate an analysis result vector;

[0015] S6. Construct a loss function based on the analysis result vector and the target label vector, and perform backpropagation to optimize the weight parameters in the deep dense residual network model;

[0016] S7. Deploy the optimized deep dense residual network model and heterogeneous feature fusion and completion algorithm module, and repeat steps S1 to S6 for the newly added heterogeneous raw data.

[0017] This paper proposes a big data analysis method based on a deep dense residual network and a heterogeneous feature fusion and completion algorithm. This method effectively addresses existing issues such as inconsistent field semantics, severe feature loss, crude fusion methods, and insufficient feature expression when processing multi-source heterogeneous data. Standardized preprocessing and unified encoding of heterogeneous raw data ensure consistency in numerical scale and semantic space across fields from different sources. Feature concatenation and vector matrix construction help preserve the structured information of each field type. The fusion and completion process is completed through three stages: feature dimension alignment, missing interpolation, and semantic mapping. This ensures that synonymous fields are aligned, missing values ​​are automatically filled, and heterogeneous homogeneous features are efficiently fused, significantly improving data availability and model input quality. The deep dense residual network architecture further enhances the expressive depth of feature extraction. The combined action of multiple densely connected modules and residual connection modules enables cross-layer information reuse and enhanced nonlinear feature transformation capabilities, enabling the model to effectively learn complex semantic dependencies when processing high-dimensional fused data. Finally, analysis results are generated through output layer mapping, and weight parameter optimization is performed in conjunction with supervisory signals to complete the model training loop. This method has high adaptability, strong generalization and good structural flexibility, and can be widely applied to large-scale intelligent analysis tasks driven by multi-source heterogeneous data.

[0018] Optionally, the S2 specifically includes:

[0019] S21, perform context-aware word embedding processing on the categorical fields in the standardized data set, define the context-aware embedding function, and transform the field value x c With the context vector z c Mapped to category dense vector v c :

[0020] S22, perform segmented normalization on the numeric fields in the standardized data set, and convert the original field value x n Convert to scaled value

[0021]

[0022] Among them, μ n is the mean of the numeric field in the standardized data set, σ n is the standard deviation of the numerical field in the standardized data set, λ is the segmented normalization threshold constant, and the scaled value Input numerical dense mapping function to generate numerical dense vector v n :

[0023] S23, use the cross-modal fusion method to extract unified feature representation for the unstructured fields in the standardized data set, define the multimodal fusion function, and transform the text feature vector v tand the image feature vector v v Mapped to fusion vector v m :

[0024] S24. Concatenate the category dense vector, the numerical dense vector, and the fusion vector and input them into the nonlinear transformation function to generate a preliminary eigenvector matrix V:

[0025] V=g([v c ‖v n ‖v m ]);

[0026] Where g(·) is a nonlinear transformation function and ‖ represents the column concatenation operation.

[0027] By performing embedded coding and feature extraction operations on different types of fields in the standardized data set, a dense vector of uniform dimension is constructed, and feature splicing is performed to form a preliminary feature vector matrix, thereby achieving effective structuring and vectorization of heterogeneous raw data. This step can significantly improve the model's adaptability to data from different sources and reduce the information loss problem caused by inconsistent field types in traditional methods. Compared with directly inputting structured and unstructured fields into the model, the preliminary feature vector matrix generated in this step has dimensional consistency and information retention integrity, providing a stable and standardized input basis for subsequent fusion completion and deep modeling, avoiding the subjective bias and poor transferability caused by artificial rule feature engineering.

[0028] Optionally, the heterogeneous feature fusion and completion algorithm specifically includes:

[0029] Performing a feature dimension alignment operation on the preliminary feature vector matrix, rearranging the feature dimensions according to predefined field mapping rules so that feature dimensions with the same semantics are consistent across different data sources, and generating an aligned feature vector matrix, where the feature dimensions are dense representation vectors in the preliminary feature vector matrix;

[0030] Perform missing feature interpolation and completion operations on the aligned feature vector matrix to identify the location of the missing features. A kernel-weighted interpolation strategy is constructed based on the similarity between the feature dimension and multiple neighboring feature dimensions. The kernel similarity weight is calculated and the corresponding eigenvalues ​​of the neighboring feature dimensions are weightedly fused to generate a complete dense feature representation.

[0031] Perform semantic isomorphic feature mapping on the heterogeneous and isomorphic complete dense features in the complete dense feature representation, map the semantic isomorphic features to a shared embedding mapping function and superimpose residual connections to obtain a set of semantic projection vectors;

[0032] A multi-channel attention fusion mechanism is applied to the set of semantic projection vectors, weighted coefficients are calculated in multiple attention channels respectively, fusion operations are performed, and the fused feature vector matrix is ​​output.

[0033] The heterogeneous feature fusion and completion algorithm achieves deep collaboration across multi-source feature dimensions through a three-stage processing mechanism. First, the feature dimension alignment step performs structural position rearrangement of semantically equivalent fields in different data sources based on predefined field mapping rules to ensure consistency of feature dimensions. Second, vector interpolation methods are used to numerically fill in missing fields to construct a complete dense feature representation, effectively alleviating training obstacles caused by missing data. Finally, a semantically homogeneous feature mapping operation is used to encode heterogeneous data into a shared embedding space, and cross-channel fusion is combined with an attention mechanism to achieve unified expression of information at the semantic level. The overall algorithm enhances the model's robustness and expressiveness for heterogeneous data, providing key support for unified modeling in complex data scenarios.

[0034] Optionally, the S3 specifically includes:

[0035] S31, constructing a feature dimension transformation mapping matrix based on the preliminary feature vector matrix V according to the predefined field mapping rules, performing a position rearrangement operation on the feature dimension, and generating an aligned feature vector matrix The feature dimension is the dense representation vector in the preliminary feature vector matrix:

[0036]

[0037] Among them, P is the feature dimension transformation mapping matrix;

[0038] S32. Identify the alignment feature vector matrix The position of the missing features in , construct the missing mask vector M:

[0039]

[0040] Among them, M i is the i-th missing mask vector; based on the similarity between the feature dimension and multiple neighboring feature dimensions, a kernel weighted interpolation strategy is constructed to calculate the kernel similarity weight and weightedly fuse the corresponding eigenvalues ​​of the neighboring feature dimensions to fill in the missing features and generate a complete dense feature representation

[0041]

[0042] in, is the i-th complete dense feature representation after missing completion, k is the number of neighbor feature dimensions, j is the index of the neighbor feature dimension, and the kernel function is an exponential kernel based on cosine similarity:

[0043]

[0044] Where exp(·) is an exponential function, and γ is a hyperparameter of the kernel function, which is used to control the sensitivity of the similarity calculation;

[0045] S33, for complete dense feature representation The complete dense features belonging to heterogeneous isomorphism in construct heterogeneous isomorphic feature vectors Where n is the number of heterogeneous isomorphic feature vectors. The shared embedding mapping function is input to perform semantic mapping to obtain a set of semantic vectors:

[0046]

[0047] Among them, f sem (·) is the shared embedding mapping function, and the semantic vector set after mapping Apply multi-channel attention mechanism and calculate weighted coefficient α i , and generate the fused feature vector v f :

[0048]

[0049] Among them, the attention weight α i Defined as:

[0050]

[0051] Among them, e i is the attention score of the i-th heterogeneous homomorphic feature vector:

[0052]

[0053] Among them, W, w, b are trainable parameters, tanh(·) is the activation function, and the final output is the fusion feature vector matrix v f .

[0054] By performing a systematic fusion and completion process on the preliminary feature vector matrix, the controllability and adaptability of the fusion mechanism are refined. Explicitly performing feature dimension alignment operations ensures the consistency and computability of the model input structure; interpolation and completion of missing features effectively avoid information fragmentation caused by missing fields and improve data integrity; the combined effect of semantic isomorphic feature mapping and attention fusion operations not only unifies the semantic expression methods across data sources, but also enhances the model's ability to focus on key fields, helping to retain discriminative features and suppress redundant feature inputs. While maintaining the integrity of the global feature structure, the overall process improves the efficiency of local information extraction, making subsequent deep modeling more targeted and stable in representation.

[0055] Optionally, the deep dense residual network model is composed of multiple groups of dense connection modules and residual connection modules:

[0056] In each densely connected module, the fused feature vector matrix is ​​processed as input features through the convolution layer. The size of the convolution kernel is adaptively adjusted based on the local statistical characteristics of the input features. The adaptively adjusted convolution kernel is used to perform the convolution operation, extract the key information from the input features, and pass the convolution result to the next layer.

[0057] In the residual connection module, the input features are fused with the features of the previous layer through the additive fusion layer. A gating mechanism is introduced in the fusion process to adjust the information transmission intensity by calculating the gating weight and control the information flow ratio between different layers. The gating weight is the average value of the input features and is used to adjust the intensity of input feature transmission.

[0058] In the output layer of the deep dense residual network model, an adaptive clustering algorithm is executed to cluster the output features, and the features are weighted according to the clustering results. The weighted features obtained after clustering are passed to the final output layer to generate deep semantic feature representation.

[0059] The deep dense residual network model structure integrates multiple groups of dense connection modules and residual connection modules, which enhances the model's ability to express high-dimensional fusion features and information transmission efficiency. The dense connection module forms a deep feature flow through continuous feature splicing and nonlinear activation operations, avoiding the decline in learning ability of traditional models due to sparse feature channels; the residual connection module realizes the direct transmission of cross-layer feature paths, alleviating the gradient disappearance and degradation problems in deep models. Compared with traditional shallow convolution or single-layer perception networks, this model can deeply explore the long-term dependencies between complex features while ensuring that the parameters are controllable, so that the model has a higher learning ability for multi-dimensional, highly coupled input features, and provides a discriminative deep semantic feature representation for the final analysis output.

[0060] Optionally, the S4 specifically includes:

[0061] S41, fusion feature vector matrix v f Input as input features to the deep dense residual network model;

[0062] S42. In each densely connected module, the input features are processed by the convolution layer, and the size of the convolution kernel k is adaptively adjusted based on the local statistical characteristics of the input features:

[0063]

[0064] in, is the fusion feature vector matrix v f The i-th eigenvector in , is the fusion feature vector matrix v f , N is the input feature dimension, α and β are adjustable hyperparameters used to control the dynamic adjustment of the convolution kernel size, δ is the minimum convolution kernel size, and the convolution operation is performed through the adaptive convolution kernel size to generate the convolution output feature map;

[0065] S43, performing batch normalization on the convolution output feature map, and after batch normalization, using the ReLU function for activation to generate activation input features;

[0066] S44. In the residual connection module, a gating mechanism is introduced to add the activation input feature with the activation input feature of the previous layer through the additive fusion layer to generate the residual connection input feature;

[0067] S45. In the output layer of the deep dense residual network model, an adaptive clustering algorithm is executed to cluster and weight the residual connection input features to generate a deep semantic feature representation.

[0068] By refining the processing process of the deep dense residual network and clarifying the functional path of each sub-module, the convolution operation is combined with an adaptive convolution kernel size adjustment strategy, so that the model can flexibly adjust the receptive field size according to the local changes of the input features, enhancing the local adaptability of feature capture; batch normalization and nonlinear activation processing are performed after the convolution output to improve the numerical stability of the model and its ability to express nonlinear features; a gating mechanism is introduced in the residual connection module to adjust the strength of information fusion between the front and back layers, so that the network can dynamically screen effective features and suppress redundant content; the deep semantic features finally output have a sense of structural hierarchy and information density, providing a high-quality representation basis for the subsequent analysis results.

[0069] Optionally, the S5 specifically includes:

[0070] S51, represent the deep semantic features v deep Input to a fully connected output layer, wherein the fully connected output layer includes a weight matrix and a bias vector;

[0071] S52, perform matrix multiplication and bias addition operations, and calculate the analysis result vector

[0072]

[0073] Among them, W o is the weight matrix of the fully connected output layer, b o is the bias vector of the fully connected output layer.

[0074] By inputting deep semantic feature representations into a fully connected output layer, a mapping pathway from semantic vectors to analysis results is established. This process improves the model's interpretability and output consistency for analysis tasks, while providing a clear error signal source for supervised training, enabling the model to have good trainability and task generalization capabilities.

[0075] Optionally, the S6 specifically includes:

[0076] S61, use the index matching module to obtain the corresponding target label vector y for the analysis result vector * , pair the analysis result vector with the target label vector to generate training sample pairs;

[0077] S62: Input the training sample pair into the loss calculation module, select and execute the loss function construction based on the task type:

[0078]

[0079] Where m is the number of training sample pairs;

[0080] S63. Based on the loss function and the analysis result vector calculation process, perform chain derivation on the weight matrix of the fully connected output layer and the bias vector of the fully connected output layer to generate a gradient expression;

[0081] S64. Take the gradient expression as input and use the set learning rate to iteratively update the weight matrix of the fully connected output layer and the bias vector of the fully connected output layer.

[0082] By pairing predictions with target labels and calculating a loss function, a mechanism for measuring the error between the model output and the desired result is established. The chain rule is used to generate gradient expressions for the output layer parameters, and the weights and biases are updated using stochastic gradient descent, completing a closed-loop training process in which the parameters gradually approach the optimal solution. This step not only ensures the mathematical differentiability and gradient controllability of model training, but also improves convergence efficiency and training stability, laying a solid foundation for effective model learning in heterogeneous data scenarios.

[0083] The beneficial effects of the present invention are:

[0084] In response to the problems existing in the process of heterogeneous multi-source data fusion modeling, such as inconsistent feature dimensions, difficulty in handling missing fields, weak semantic representation, and poor model training stability, the present invention proposes a big data analysis method that combines a deep dense residual network with a heterogeneous feature fusion and completion algorithm. Different from the traditional way of cleaning and encoding fields with static rules, the present invention first constructs a preliminary feature vector matrix through structural encoding, embedded expression and splicing operations of multiple types of fields to ensure that each type of field has a unified numerical scale and dimensional structure in the same vector space, which significantly improves the parsability and robustness of subsequent models when processing high-dimensional dense inputs. In the feature fusion stage, the present invention introduces the coordinated execution of a trinity of feature dimension alignment, missing value interpolation and completion, and semantic isomorphism mapping mechanism, which solves the training gap problem caused by inconsistent formats or data gaps in heterogeneous fields, and enables consistent expression of input features at both the structural and semantic levels.

[0085] Furthermore, the deep dense residual network model constructed by this invention integrates multiple sets of densely connected modules and residual connection modules in its structural design, effectively enhancing the model's representational capabilities and feature extraction depth. Unlike the common problems of independent inter-layer information and fixed local receptive fields in existing deep neural network structures, this invention achieves dynamic perception of local changes in input features through an adaptive convolution kernel adjustment mechanism and a batch normalization-activation combination module, making the model more adaptable when dealing with non-stationary feature distributions or cross-source inputs.

[0086] In the output layer design, the present invention adopts a fully connected mapping structure to compress the deep semantic feature representation into the target analysis space, and introduces an activation mechanism that matches the task type (such as Softmax for classification and Sigmoid for regression), which ensures the numerical stability and trainability of the model output. In the training stage, the system constructs a loss function based on the mapping relationship between the prediction results and the supervision labels, and generates a gradient expression through chain derivation, which is used to accurately update the model parameters in back propagation, thereby achieving the differentiability and controllability of the training closed loop. Compared with the problem of fixed loss function form or non-gradient structure in existing methods that are difficult to converge, the training process of the present invention is highly structured and logically closed, and can stably optimize the objective function in a complex data environment, thereby improving the convergence efficiency and accuracy performance of the analysis task.

[0087] Overall, the present invention achieves efficient mapping of heterogeneous data into a unified semantic space by constructing a four-stage process of standardized input-fusion completion-deep modeling-supervised optimization, and has good generalization ability, robustness and engineering deployment flexibility in multi-task analysis. It is particularly suitable for key business scenarios such as financial risk control, medical intelligent analysis, industrial monitoring and operation and maintenance diagnosis that rely on heterogeneous multi-source big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0089] Figure 1 This is an overall flow chart of the analysis method of a dense model applied to big data proposed by the present invention;

[0090] Figure 2 This is a schematic diagram of the structure of a deep dense residual network model for applying a dense model proposed in the present invention to the analysis method of big data. DETAILED DESCRIPTION

[0091] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0092] refer to Figure 1-2 , a method for analyzing big data using a dense model, including the following steps:

[0093] S1, collect heterogeneous raw data, perform preprocessing, and generate standardized data sets;

[0094] S2. Extract features from different types of fields in the standardized data set to obtain dense representation vectors, and perform feature concatenation to form a preliminary feature vector matrix;

[0095] S3. Input the preliminary feature vector matrix into the heterogeneous feature fusion and completion algorithm, and perform feature dimension alignment, missing feature interpolation completion, and semantic homogeneous feature mapping operations in sequence to generate a fused feature vector matrix;

[0096] S4. Inputting the fused feature vector matrix into a deep dense residual network model, wherein the deep dense residual network model is composed of multiple groups of dense connection modules and residual connection modules. The dense connection modules perform continuous feature splicing, convolution transformation and nonlinear activation operations, and the residual connection modules perform cross-layer feature superposition and output deep semantic feature representation;

[0097] S5. Input the deep semantic feature representation into the fully connected output layer, perform matrix multiplication and bias addition operations, and generate an analysis result vector;

[0098] S6. Construct a loss function based on the analysis result vector and the target label vector, and perform backpropagation to optimize the weight parameters in the deep dense residual network model;

[0099] S7. Deploy the optimized deep dense residual network model and heterogeneous feature fusion and completion algorithm module, and repeat steps S1 to S6 for the newly added heterogeneous raw data.

[0100] In this embodiment, S2 specifically includes:

[0101] S21, perform context-aware word embedding processing on the categorical fields in the standardized data set, define the context-aware embedding function, and transform the field value x c With the context vector z c Mapped to category dense vector v c :

[0102] S22, perform segmented normalization on the numeric fields in the standardized data set, and convert the original field value x n Convert to scaled value

[0103]

[0104] Among them, μ n is the mean of the numeric field in the standardized data set, σ n is the standard deviation of the numerical field in the standardized data set, λ is the segmented normalization threshold constant, and the scaled value Input numerical dense mapping function to generate numerical dense vector v n :

[0105] S23, use the cross-modal fusion method to extract unified feature representation for the unstructured fields in the standardized data set, define the multimodal fusion function, and transform the text feature vector v t and the image feature vector v v Mapped to fusion vector v m :

[0106] S24. Concatenate the category dense vector, the numerical dense vector, and the fusion vector and input them into the nonlinear transformation function to generate a preliminary eigenvector matrix V:

[0107] V=g([v c ‖v n ‖v m ]);

[0108] Where g(·) is a nonlinear transformation function and ‖ represents the column concatenation operation.

[0109] In this embodiment, the heterogeneous feature fusion and completion algorithm specifically includes:

[0110] Performing a feature dimension alignment operation on the preliminary feature vector matrix, rearranging the feature dimensions according to predefined field mapping rules so that feature dimensions with the same semantics are consistent across different data sources, and generating an aligned feature vector matrix, where the feature dimensions are dense representation vectors in the preliminary feature vector matrix;

[0111] Perform missing feature interpolation and completion operations on the aligned feature vector matrix to identify the location of the missing features. A kernel-weighted interpolation strategy is constructed based on the similarity between the feature dimension and multiple neighboring feature dimensions. The kernel similarity weight is calculated and the corresponding eigenvalues ​​of the neighboring feature dimensions are weightedly fused to generate a complete dense feature representation.

[0112] Perform semantic isomorphic feature mapping on the heterogeneous and isomorphic complete dense features in the complete dense feature representation, map the semantic isomorphic features to a shared embedding mapping function and superimpose residual connections to obtain a set of semantic projection vectors;

[0113] A multi-channel attention fusion mechanism is applied to the set of semantic projection vectors, weighted coefficients are calculated in multiple attention channels respectively, fusion operations are performed, and the fused feature vector matrix is ​​output.

[0114] In this embodiment, S3 specifically includes:

[0115] S31, constructing a feature dimension transformation mapping matrix based on the preliminary feature vector matrix V according to the predefined field mapping rules, performing a position rearrangement operation on the feature dimension, and generating an aligned feature vector matrix The feature dimension is the dense representation vector in the preliminary feature vector matrix:

[0116]

[0117] Among them, P is the feature dimension transformation mapping matrix;

[0118] S32. Identify the alignment feature vector matrix The position of the missing features in , construct the missing mask vector M:

[0119]

[0120] Among them, M i is the i-th missing mask vector; based on the similarity between the feature dimension and multiple neighboring feature dimensions, a kernel weighted interpolation strategy is constructed to calculate the kernel similarity weight and weightedly fuse the corresponding eigenvalues ​​of the neighboring feature dimensions to fill in the missing features and generate a complete dense feature representation

[0121]

[0122] in, is the i-th complete dense feature representation after missing completion, k is the number of neighbor feature dimensions, j is the index of the neighbor feature dimension, and the kernel function is an exponential kernel based on cosine similarity:

[0123]

[0124] Where exp(·) is an exponential function, and γ is a hyperparameter of the kernel function, which is used to control the sensitivity of the similarity calculation;

[0125] S33, for complete dense feature representation The complete dense features belonging to heterogeneous isomorphism in construct heterogeneous isomorphic feature vectors Where n is the number of heterogeneous isomorphic feature vectors. The shared embedding mapping function is input to perform semantic mapping to obtain a set of semantic vectors:

[0126]

[0127] Among them, f sem (·) is the shared embedding mapping function, and the semantic vector set after mapping Apply multi-channel attention mechanism and calculate weighted coefficient α i , and generate the fused feature vector v f :

[0128]

[0129] Among them, the attention weight α i Defined as:

[0130]

[0131] Among them, e i is the attention score of the i-th heterogeneous homomorphic feature vector:

[0132]

[0133] Among them, W, w, b are trainable parameters, tanh(·) is the activation function, and the final output is the fusion feature vector matrix v f .

[0134] In this embodiment, the deep dense residual network model is composed of multiple groups of dense connection modules and residual connection modules:

[0135] In each densely connected module, the fused feature vector matrix is ​​processed as input features through the convolution layer. The size of the convolution kernel is adaptively adjusted based on the local statistical characteristics of the input features. The adaptively adjusted convolution kernel is used to perform the convolution operation, extract the key information from the input features, and pass the convolution result to the next layer.

[0136] In the residual connection module, the input features are fused with the features of the previous layer through the additive fusion layer. A gating mechanism is introduced in the fusion process to adjust the information transmission intensity by calculating the gating weight and control the information flow ratio between different layers. The gating weight is the average value of the input features and is used to adjust the intensity of input feature transmission.

[0137] In the output layer of the deep dense residual network model, an adaptive clustering algorithm is executed to cluster the output features, and the features are weighted according to the clustering results. The weighted features obtained after clustering are passed to the final output layer to generate deep semantic feature representation.

[0138] In this embodiment, the S4 specifically includes:

[0139] S41, fusion feature vector matrix v f Input as input features to the deep dense residual network model;

[0140] S42. In each densely connected module, the input features are processed by the convolution layer, and the size of the convolution kernel k is adaptively adjusted based on the local statistical characteristics of the input features:

[0141]

[0142] in, is the fusion feature vector matrix v f The i-th eigenvector in , is the fusion feature vector matrix v f , N is the input feature dimension, α and β are adjustable hyperparameters used to control the dynamic adjustment of the convolution kernel size, δ is the minimum convolution kernel size, and the convolution operation is performed through the adaptive convolution kernel size to generate the convolution output feature map;

[0143] S43, performing batch normalization on the convolution output feature map, and after batch normalization, using the ReLU function for activation to generate activation input features;

[0144] S44. In the residual connection module, a gating mechanism is introduced to add the activation input feature with the activation input feature of the previous layer through the additive fusion layer to generate the residual connection input feature;

[0145] S45. In the output layer of the deep dense residual network model, an adaptive clustering algorithm is executed to cluster and weight the residual connection input features to generate a deep semantic feature representation.

[0146] In this embodiment, the S5 specifically includes:

[0147] S51, represent the deep semantic features v deepInput to a fully connected output layer, wherein the fully connected output layer includes a weight matrix and a bias vector;

[0148] S52, perform matrix multiplication and bias addition operations, and calculate the analysis result vector

[0149]

[0150] Among them, W o is the weight matrix of the fully connected output layer, b o is the bias vector of the fully connected output layer.

[0151] In this embodiment, S6 specifically includes:

[0152] S61, use the index matching module to obtain the corresponding target label vector y for the analysis result vector * , pair the analysis result vector with the target label vector to generate training sample pairs;

[0153] S62: Input the training sample pair into the loss calculation module, select and execute the loss function construction based on the task type:

[0154]

[0155] Where m is the number of training sample pairs;

[0156] S63. Based on the loss function and the analysis result vector calculation process, perform chain derivation on the weight matrix of the fully connected output layer and the bias vector of the fully connected output layer to generate a gradient expression;

[0157] S64. Take the gradient expression as input and use the set learning rate to iteratively update the weight matrix of the fully connected output layer and the bias vector of the fully connected output layer.

[0158] Example 1:

[0159] To verify the feasibility and effectiveness of this invention in real-world scenarios, from August to October 2024, a provincial commercial bank successfully deployed an intelligent data modeling system using a dense model analysis method proposed in this invention to address the challenges of heterogeneous big data integration and low analytical efficiency in its customer operations system. The system utilized over 130 fields, spanning numerical, categorical, time series, and Boolean data types, exhibiting typical big data characteristics such as high dimensionality inconsistency, significant missing data, and non-standard synonymous fields.

[0160] During the implementation process, the system first performs standardization and unified encoding on all data fields, uses word embedding and continuous value normalization methods to generate preliminary feature vectors, then completes feature dimension alignment through the field mapping table, and uses an interpolation strategy based on similar samples to fill in missing values. The fused features are then input into a deep dense residual network, where multi-level semantic features are extracted through dense connections, and cross-layer information aggregation is enhanced through residual paths to generate deep semantic feature vectors. Finally, the model outputs prediction scores based on different business objectives to achieve user stratification identification and intelligent recommendations. The system uses GPU cluster training, processes an average of more than 120,000 samples per day, and keeps the inference efficiency within 500 milliseconds.

[0161] To evaluate the effectiveness of the method of the present invention, the project team compared the analysis accuracy and business indicator performance of the original shallow logic model and the system of the present invention in a customer marketing recommendation scenario. The results are shown in the following table:

[0162] Table 1 Comparative data of diagnosis before and after implementation of the present invention

[0163]

[0164]

[0165] As can be seen from the data in Table 1, the method of the present invention demonstrates significant advantages in the accuracy and efficiency of the recommendation system. First, the recommendation response rate increased from the original 6.1% to 14.2%, an increase of 132.8%. This shows that the system can successfully match recommendations to more user groups and is more accurate in matching recommended content with user needs. In terms of conversion rate, the original model was 3.0%, while the method of the present invention increased it to 8.1%, an increase of 170.0%. This improvement shows that the method of the present invention has a significant effect in guiding users to take action and increasing their purchase intention of recommended products.

[0166] In terms of its ability to handle missing fields, the original model's availability when faced with missing fields was only below 20%. However, the proposed method can still ensure system operation even with higher missing rates, effectively handling over 40% of missing fields. This demonstrates that the proposed method has a strong advantage in processing and completing heterogeneous data, and is particularly well-suited for the high-missing data problem common in big data analysis. Further analysis shows that the AUC value increased from 0.74 to 0.89, a 20.3% increase. This change demonstrates that the proposed method has significantly improved the model's ability to discriminate between various types of samples, enabling more accurate identification of user interests and needs.

[0167] The F1 score of our method improved from 0.41 to 0.68, a 65.9% improvement. This demonstrates a balanced improvement in both precision and recall, particularly when processing multi-dimensional features, long-tail data, and diverse user groups. The improved Top-3 accuracy also demonstrates the model's enhanced ability to rank multiple recommendation options, enabling more personalized recommendations for users.

[0168] The system takes a relatively long time to train, with the average training time increasing from 2.6 hours to 4.7 hours. However, this increase is achieved in exchange for higher accuracy and deeper model training. Therefore, in actual deployment, it still meets user requirements for accuracy and performance. Inference latency is controlled within 510ms. Although slightly higher than the 420ms of the original model, this latency is still within an acceptable range and can meet the requirements of real-time recommendations. In particular, the system's real-time inference capabilities and responsiveness are guaranteed in practical applications.

[0169] In summary, the present invention significantly improves the accuracy and robustness of the recommendation system by constructing a complete data flow and optimization path from feature extraction, fusion completion to deep modeling and output. The system not only solves the technical bottlenecks in traditional methods in heterogeneous data fusion, feature processing, missing completion, etc., but also enhances the modeling capabilities of high-dimensional features, cross-source data and complex user behavior patterns, providing efficient and accurate solutions for various business scenarios. Especially for industries such as finance, e-commerce, and advertising that require accurate recommendations, the method of the present invention has strong application potential and wide promotion value, and can support large-scale personalized recommendation services.

[0170] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for analyzing big data using a dense model, characterized in that: The steps include: S1, collect heterogeneous raw data, perform preprocessing, and generate standardized data sets; S2. Extract features from different types of fields in the standardized data set to obtain dense representation vectors, and perform feature concatenation to form a preliminary feature vector matrix; S3. Input the preliminary feature vector matrix into the heterogeneous feature fusion and completion algorithm, and perform feature dimension alignment, missing feature interpolation completion, and semantic homogeneous feature mapping operations in sequence to generate a fused feature vector matrix; S4. Inputting the fused feature vector matrix into a deep dense residual network model, wherein the deep dense residual network model is composed of multiple groups of dense connection modules and residual connection modules. The dense connection modules perform continuous feature splicing, convolution transformation and nonlinear activation operations, and the residual connection modules perform cross-layer feature superposition and output deep semantic feature representation; S5. Input the deep semantic feature representation into the fully connected output layer, perform matrix multiplication and bias addition operations, and generate an analysis result vector; S6. Construct a loss function based on the analysis result vector and the target label vector, and perform backpropagation to optimize the weight parameters in the deep dense residual network model; S7. Deploy the optimized deep dense residual network model and heterogeneous feature fusion and completion algorithm module, and repeat steps S1 to S6 for the newly added heterogeneous raw data.

2. The method for analyzing big data using a dense model according to claim 1, characterized in that: The S2 specifically includes: S21, perform context-aware word embedding processing on the categorical fields in the standardized data set, define the context-aware embedding function, and transform the field value x c With the context vector z c Mapped to category dense vector v c : S22, perform segmented normalization on the numeric fields in the standardized data set, and convert the original field value x n Convert to scaled value Among them, μ n is the mean of the numeric field in the standardized data set, σ n is the standard deviation of the numerical field in the standardized data set, λ is the segmented normalization threshold constant, and the scaled value Input numerical dense mapping function to generate numerical dense vector v n : S23, use the cross-modal fusion method to extract unified feature representation for the unstructured fields in the standardized data set, define the multimodal fusion function, and transform the text feature vector v t and the image feature vector v v Mapped to fusion vector v m : S24. Concatenate the category dense vector, the numerical dense vector, and the fusion vector and input them into the nonlinear transformation function to generate a preliminary eigenvector matrix V: V=g([v c ‖v n ‖v m ]); Where g(·) is a nonlinear transformation function and ‖ represents the column concatenation operation.

3. The method for analyzing big data using a dense model according to claim 1, characterized in that: The heterogeneous feature fusion and completion algorithm specifically includes: Performing a feature dimension alignment operation on the preliminary feature vector matrix, rearranging the feature dimensions according to predefined field mapping rules so that feature dimensions with the same semantics are consistent across different data sources, and generating an aligned feature vector matrix, where the feature dimensions are dense representation vectors in the preliminary feature vector matrix; Perform missing feature interpolation and completion operations on the aligned feature vector matrix to identify the location of the missing features. A kernel-weighted interpolation strategy is constructed based on the similarity between the feature dimension and multiple neighboring feature dimensions. The kernel similarity weight is calculated and the corresponding eigenvalues ​​of the neighboring feature dimensions are weightedly fused to generate a complete dense feature representation. Perform semantic isomorphic feature mapping on the heterogeneous and isomorphic complete dense features in the complete dense feature representation, map the semantic isomorphic features to a shared embedding mapping function and superimpose residual connections to obtain a set of semantic projection vectors; A multi-channel attention fusion mechanism is applied to the set of semantic projection vectors, weighted coefficients are calculated in multiple attention channels respectively, fusion operations are performed, and the fused feature vector matrix is ​​output.

4. The method for analyzing big data using a dense model according to claim 1, wherein: The S3 specifically includes: S31, constructing a feature dimension transformation mapping matrix based on the preliminary feature vector matrix V according to the predefined field mapping rules, performing a position rearrangement operation on the feature dimension, and generating an aligned feature vector matrix The feature dimension is the dense representation vector in the preliminary feature vector matrix: Among them, P is the feature dimension transformation mapping matrix; S32. Identify the alignment feature vector matrix The position of the missing features in , construct the missing mask vector M: Among them, M i is the i-th missing mask vector; based on the similarity between the feature dimension and multiple neighboring feature dimensions, a kernel weighted interpolation strategy is constructed to calculate the kernel similarity weight and weightedly fuse the corresponding eigenvalues ​​of the neighboring feature dimensions to fill in the missing features and generate a complete dense feature representation in, is the i-th complete dense feature representation after missing completion, k is the number of neighbor feature dimensions, j is the index of the neighbor feature dimension, and the kernel function is an exponential kernel based on cosine similarity: Where exp(·) is an exponential function, and γ is a hyperparameter of the kernel function, which is used to control the sensitivity of the similarity calculation; S33, for complete dense feature representation The complete dense features belonging to heterogeneous isomorphism in construct heterogeneous isomorphic feature vectors Where n is the number of heterogeneous isomorphic feature vectors. The shared embedding mapping function is input to perform semantic mapping to obtain a set of semantic vectors: Among them, f sem (·) is the shared embedding mapping function, and the semantic vector set after mapping Apply multi-channel attention mechanism and calculate weighted coefficient α i , and generate the fusion feature vector v f : Among them, the attention weight α i Defined as: Among them, e i is the attention score of the i-th heterogeneous homomorphic feature vector: Among them, W, w, b are trainable parameters, tanh(·) is the activation function, and the final output is the fusion feature vector matrix v f .

5. The method for analyzing big data using a dense model according to claim 1, characterized in that: The deep dense residual network model consists of multiple groups of dense connection modules and residual connection modules: In each densely connected module, the fused feature vector matrix is ​​processed as input features through the convolution layer. The size of the convolution kernel is adaptively adjusted based on the local statistical characteristics of the input features. The adaptively adjusted convolution kernel is used to perform the convolution operation, extract the key information from the input features, and pass the convolution result to the next layer. In the residual connection module, the input features are fused with the features of the previous layer through the additive fusion layer. A gating mechanism is introduced in the fusion process to adjust the information transmission intensity by calculating the gating weight and control the information flow ratio between different layers. The gating weight is the average value of the input features and is used to adjust the intensity of input feature transmission. In the output layer of the deep dense residual network model, an adaptive clustering algorithm is executed to cluster the output features, and the features are weighted according to the clustering results. The weighted features obtained after clustering are passed to the final output layer to generate deep semantic feature representation.

6. The method for analyzing big data using a dense model according to claim 1, characterized in that: The S4 specifically includes: S41, fusion feature vector matrix v f Input as input features to the deep dense residual network model; S42. In each densely connected module, the input features are processed by the convolution layer, and the size of the convolution kernel k is adaptively adjusted based on the local statistical characteristics of the input features: in, is the fusion feature vector matrix v f The i-th eigenvector in , is the fusion feature vector matrix v f , N is the input feature dimension, α and β are adjustable hyperparameters used to control the dynamic adjustment of the convolution kernel size, δ is the minimum convolution kernel size, and the convolution operation is performed through the adaptive convolution kernel size to generate the convolution output feature map; S43, performing batch normalization on the convolution output feature map, and after batch normalization, using the ReLU function for activation to generate activation input features; S44. In the residual connection module, a gating mechanism is introduced to add the activation input feature with the activation input feature of the previous layer through the additive fusion layer to generate the residual connection input feature; S45. In the output layer of the deep dense residual network model, an adaptive clustering algorithm is executed to cluster and weight the residual connection input features to generate a deep semantic feature representation.

7. The method for analyzing big data using a dense model according to claim 1, characterized in that: The S5 specifically includes: S51, represent the deep semantic features v deep Input to a fully connected output layer, wherein the fully connected output layer includes a weight matrix and a bias vector; S52, perform matrix multiplication and bias addition operations, and calculate the analysis result vector Among them, W o is the weight matrix of the fully connected output layer, b o is the bias vector of the fully connected output layer.

8. The method for analyzing big data using a dense model according to claim 1, wherein: The S6 specifically includes: S61, use the index matching module to obtain the corresponding target label vector y for the analysis result vector * , pair the analysis result vector with the target label vector to generate training sample pairs; S62: Input the training sample pair into the loss calculation module, select and execute the loss function construction based on the task type: Where m is the number of training sample pairs; S63. Based on the loss function and the analysis result vector calculation process, perform chain derivation on the weight matrix of the fully connected output layer and the bias vector of the fully connected output layer to generate a gradient expression; S64. Take the gradient expression as input and use the set learning rate to iteratively update the weight matrix of the fully connected output layer and the bias vector of the fully connected output layer.

Images