Self-encoding neural network training method based on feature refinement and customized large model
Through the self-coding neural network training method based on feature refinement, the problems of excessive complexity and feature loss in high-dimensional industrial data processing are solved, and the adaptation to natural language processing in industrial equipment and the improvement of feature extraction accuracy is achieved.
Patent Information
- Application Number
- CN202510070656.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-16
- Publication Date
- 2025-04-25
AI Technical Summary
When processing high-dimensional industrial data, the prior art can easily lead to excessive complexity of the model or loss of features, affecting model performance, and lack of specially customized large models, making it difficult to adapt to the natural language processing needs related to industrial equipment.
The self-encoding neural network training method based on feature refinement is adopted, and the high-dimensional data is mapped to the low-dimensional feature space through the encoder. The feature adjustment module recursively optimizes the feature weight, enhances important features and weakens redundant or noise features, and the decoder reconstructs the features after dimensionality reduction back to the original space.
The model's ability to capture key features is strengthened, ensuring that valuable data information is not lost in the process of reducing dimensions, improving the accuracy of feature extraction and the simplicity of the model, and adapting to the natural language processing needs in the industrial equipment field.
Smart Images
Figure CN119886250A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a self-encoding neural network training method based on feature refinement and a customized large model. Background Art
[0002] At present, there are still many problems in intelligent dialogue systems and related technical fields, which make it difficult to meet the increasingly complex user needs and application scenarios. With the rapid development of artificial intelligence technology, intelligent dialogue systems are widely used in customer service, education, medical care, finance and other fields. For example, the digitalization and intelligentization process in the field of industrial equipment continues to advance, and the automated operation, remote monitoring and fault diagnosis of equipment have become industry trends. However, the data of industrial equipment such as operation manuals, maintenance records, fault logs, etc. are numerous and complex, involving not only real-time data of machine operation, but also a large amount of natural language text processing, such as equipment fault reports, technical documents, etc. How to make the model effectively understand this data and provide intelligent decision support and fault solutions has become a major challenge in current technology.
[0003] However, in the traditional neural network training process, when processing high-dimensional industrial data, it is often easy to cause the model complexity to be too high or feature loss, which affects the performance of the model. The use of feature selection and dimensionality reduction methods has also failed to effectively solve the problem of feature redundancy and noise. In addition, traditional large models are usually general-purpose and perform poorly in specific industry applications. The existing technology lacks specialized customized large models in the field of industrial equipment, resulting in a weak understanding of the professional knowledge in this field and difficulty in adapting to the natural language processing needs related to industrial equipment.
[0004] In view of this, it is necessary to provide an autoencoder neural network training method based on feature refinement and a customized large model to overcome the above defects. Summary of the invention
[0005] The purpose of the present invention is to provide an autoencoder neural network training method based on feature refinement and a customized large model, aiming to solve the problem that the existing technology often easily leads to excessive model complexity or feature loss when processing high-dimensional industrial data, thereby affecting the performance of the model, strengthen the model's ability to capture key features, ensure that valuable data information is not lost in the process of reducing the dimension during model training, and improve the accuracy of feature extraction and the simplicity of the model.
[0006] In order to achieve the above object, the present invention provides an autoencoder neural network training method based on feature refinement for Transformer network training, comprising:
[0007] Step S11: The encoder uses a multi-layer nonlinear mapping structure to map the high-dimensional data in the input training data to the initial low-dimensional feature space;
[0008] Step S12: After the low-dimensional features are generated, the feature adjustment module automatically generates feature weights according to the feature importance in the current feature space; wherein the feature adjustment module assigns the same initial weight to all low-dimensional features during initialization;
[0009] Step S13: The feature adjustment module recursively optimizes the initially generated low-dimensional features. In each round of iteration, the feature adjustment module adjusts the weights of each feature according to the performance of the previous round of features, gradually enhances the features with important influence, and gradually weakens the redundant or noise features;
[0010] Step S14: The decoder remaps the low-dimensional features back to the high-dimensional space, and then calculates the loss function to determine whether the preset stop iteration condition is met. If the result is no, the encoder maps the high-order data after the previous round of iteration to the low-dimensional feature space again for the next round of training; if the result is yes, the training ends.
[0011] In a preferred embodiment, in step S11, the data input to the autoencoder neural network is X r , the encoder maps the high-dimensional data to the initial low-dimensional feature space, which is expressed as:
[0012] Z r =Sig enc (W r X r +b r ),
[0013] In the formula, Z r represents the initial low-dimensional features, W r is the weight matrix of the encoder, b r is the encoder bias vector, Sig enc () is the multi-layer Sigmoid activation function of the encoder.
[0014] In a preferred embodiment, in step S12, the calculation method of the initial weight matrix is expressed as:
[0015] A r =diag(α r ),
[0016] In the formula, A r is the initial weight matrix; α r is the feature weight vector, α r Each element α in r,i The same value is used at the time of initialization to indicate that all features have the same importance in the initial stage; diag() is a function for extracting the diagonal elements of the matrix;
[0017] Among them, the adjusted features can be expressed as:
[0018] Z ′ r =A r Z r ,
[0019] In the formula, Z ′ r is the feature representation after feature weight adjustment.
[0020] In a preferred embodiment, in step S13, it is assumed that in the tth iteration, the weight update rule is as follows:
[0021]
[0022] In the formula, represents the feature weight of the t+1th iteration, represents the feature weight of the tth iteration, η r is the learning rate, L r () is the loss function, Y r is the label data, Represents the gradient of the loss function with respect to the feature weights.
[0023] In a preferred embodiment, in step S13, the loss function is a reconstruction error loss function.
[0024] In a preferred embodiment, in step S13, n r Set to 0.01.
[0025] In a preferred embodiment, in step S14, the reconstruction process of the decoder is expressed as:
[0026] X r ′ =Sig dec (W r ′ Z r ′ +b r ′ )
[0027] Where, X r ′ is the reconstructed high-dimensional data, W r ′ is the weight matrix of the decoder, b r ′ is the bias vector of the decoder, Sig dec () is the multi-layer Sigmoid activation function of the decoder.
[0028] The present invention also provides a customized large model, which is a model based on a Transformer network, including an encoder, a decoder and a feature adjustment module; the encoder is used to map high-dimensional data in the training data to a low-dimensional feature space; the feature adjustment module is used to dynamically adjust the feature space after dimensionality reduction through recursive feature adaptive optimization, so that important features can be strengthened and secondary features are gradually weakened; the decoder is used to reconstruct the features after dimensionality reduction back to the original space to ensure the reversibility of the dimensionality reduction process.
[0029] In a preferred embodiment, the customized large model is a natural language large model in the field of industrial equipment, and the training data used in its training includes:
[0030] Equipment operation manuals and technical documents: record the use, maintenance and technical parameters of the equipment in detail, to help the customized large model understand the equipment operation specifications, extract technical points, and realize automatic document parsing and troubleshooting suggestions;
[0031] Maintenance records and repair reports: contain equipment failure descriptions, repair steps, and spare parts used to extract common failure modes and predict maintenance needs;
[0032] Fault log and alarm information: Record abnormalities in equipment operation in short text or structured data for fault mode identification and alarm optimization;
[0033] Industry standards and regulatory documents: Include compliance requirements for equipment operation, which are used to help companies ensure that equipment operations comply with regulations by parsing regulatory documents.
[0034] The autoencoding neural network training method based on feature refinement and the customized large model provided by the present invention adopt an autoencoding neural network based on feature refinement. The encoder is responsible for mapping the input high-dimensional data to a low-dimensional feature space, and the decoder is used to reconstruct the reduced-dimensional features back to the original space to ensure the reversibility of the dimensionality reduction process; the feature adjustment module dynamically adjusts the reduced-dimensional feature space through recursive feature adaptive optimization, so that important features can be strengthened and secondary features are gradually weakened, further strengthening the model's ability to capture key features, ensuring that the model does not lose valuable data information in the process of reducing the dimensionality, and improving the accuracy of feature extraction and the simplicity of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0036] Figure 1 It is a flowchart of the autoencoder neural network training method based on feature refinement;
[0037] Figure 2 This is the architecture diagram for a custom large model. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and beneficial technical effect of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described in this specification are only for explaining the present invention, not for limiting the present invention.
[0039] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0040] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0041] In an embodiment of the present invention, a method for training an autoencoder neural network based on feature refinement is provided for training a Transformer network, which reduces the dimensionality of input data while retaining important features and weakening irrelevant or redundant information. Through adaptive recursive optimization, important features can be strengthened and secondary features are gradually weakened, so that the feature representation after dimensionality reduction has better simplicity while retaining data information.
[0042] It should be noted that the model based on the Transformer network in this embodiment includes an encoder, a decoder and a feature adjustment module. The encoder is used to map the high-dimensional data in the training data to a low-dimensional feature space. The feature adjustment module is used to dynamically adjust the feature space after dimensionality reduction through recursive feature adaptive optimization, so that important features can be strengthened and secondary features are gradually weakened. The decoder is used to reconstruct the features after dimensionality reduction back to the original space to ensure the reversibility of the dimensionality reduction process.
[0043] like Figure 1 As shown, the autoencoder neural network training method based on feature refinement includes steps S11-S14.
[0044] Step S11: The encoder uses a multi-layer nonlinear mapping structure to map the high-dimensional data in the input training data to the initial low-dimensional feature space.
[0045] Specifically, let the data input to the autoencoder neural network be X r , the encoder maps the high-dimensional data to the initial low-dimensional feature space, which is expressed as:
[0046] Z r =Sig enc (W r X r +b r ),
[0047] In the formula, Z r represents the initial low-dimensional features, W r is the weight matrix of the encoder, b r is the encoder bias vector, Sig enc () is the multi-layer Sigmoid activation function of the encoder.
[0048] Step S12: After the low-dimensional features are generated, the feature adjustment module automatically generates feature weights according to the feature importance in the current feature space; wherein, the feature adjustment module assigns the same initial weights to all low-dimensional features during initialization so that they can be gradually adjusted according to feature contributions in subsequent steps.
[0049] It should be noted that when the weights of each low-dimensional feature are initialized for the first time, each feature is given the same weight, and then in subsequent iterations, the weight value of each feature is adjusted accordingly according to its contribution.
[0050] Among them, the calculation method of the initial weight matrix is expressed as:
[0051] A r =diag(α r ),
[0052] In the formula, A r is the initial weight matrix; α r is the feature weight vector, α r Each element α in r,i The same value is used at the time of initialization to indicate that all features have the same importance in the initial stage; diag() is a function for extracting the diagonal elements of the matrix.
[0053] Among them, the adjusted features can be expressed as:
[0054] Z ′ r =A r Z r ,
[0055] In the formula, Z ′ r is the feature representation after feature weight adjustment.
[0056] Step S13: The feature adjustment module recursively optimizes the initially generated low-dimensional features. In each round of iteration, the feature adjustment module adjusts the weights of each feature according to the performance of the previous round of features, gradually enhances the features with important influence, and gradually weakens the redundant or noise features.
[0057] Specifically, in the tth iteration, the weight update rule is as follows:
[0058]
[0059] In the formula, represents the feature weight of the t+1th iteration, represents the feature weight of the tth iteration, η r is the learning rate, L r () is the loss function, Y r is the label data, represents the gradient of the loss function with respect to the feature weight. Preferably, η r Set to 0.01. In one embodiment, the loss function is a reconstruction error loss function.
[0060] Step S14: The decoder remaps the low-dimensional features back to the high-dimensional space, and then calculates the loss function to determine whether the preset stop iteration condition is met. If the result is no, the encoder maps the high-order data after the previous round of iteration to the low-dimensional feature space again for the next round of training; if the result is yes, the training ends.
[0061] In order to ensure the effectiveness of the dimensionality reduction process, the decoder remaps the low-dimensional features back to the high-dimensional space to ensure that the dimensionality reduction process does not lose important information. The reconstruction process of the decoder is expressed as:
[0062] X r ′ =Sig dec (W r ′ Z r ′ +b r ′ )
[0063] Where, X r ′ is the reconstructed high-dimensional data, W r ′ is the weight matrix of the decoder, b r ′ is the bias vector of the decoder, Sig dec () is the multi-layer Sigmoid activation function of the decoder.
[0064] In this embodiment, the preset condition for stopping iteration is reaching a preset maximum number of iterations. Preferably, the preset maximum number of iterations is set to 1000 times.
[0065] The present invention also provides a customized large model. The customized large model is a model based on the Transformer network, including an encoder, a decoder and a feature adjustment module; the encoder is used to map the high-dimensional data in the training data to the low-dimensional feature space; the feature adjustment module is used to dynamically adjust the feature space after dimensionality reduction through recursive feature adaptive optimization, so that important features can be strengthened and secondary features are gradually weakened; the decoder is used to reconstruct the features after dimensionality reduction back to the original space to ensure the reversibility of the dimensionality reduction process. In this embodiment, the training process of the customized large model adopts the autoencoder neural network training method based on feature refinement as described in any of the above-mentioned implementation methods.
[0066] It should be noted that, in some embodiments, the customized large model is a natural language large model in the field of industrial equipment, and the training data used in its training includes:
[0067] Equipment operation manuals and technical documents: record the use, maintenance and technical parameters of the equipment in detail, to help the customized large model understand the equipment operation specifications, extract technical points, and realize automatic document parsing and troubleshooting suggestions;
[0068] Maintenance records and repair reports: contain equipment failure descriptions, repair steps, and spare parts used to extract common failure modes and predict maintenance needs;
[0069] Fault log and alarm information: Record abnormalities in equipment operation in short text or structured data for fault mode identification and alarm optimization;
[0070] Industry standards and regulatory documents: Include compliance requirements for equipment operation, which are used to help companies ensure that equipment operations comply with regulations by parsing regulatory documents.
[0071] Here, if Figure 2 As shown in the figure, the customized large model uses the Transformer network as the basic network architecture. The encoder maps the input features into a series of context-aware vector representations, and the decoder gradually decodes these vector representations into target features.
[0072] Specifically, the encoder includes N identical encoding layers, each encoding layer includes two sublayers, a multi-head attention mechanism sublayer and a feedforward network sublayer, and the encoder adds residual connections and normalization operations after each sublayer. The decoder includes N identical decoding layers, each decoding layer includes two multi-head attention mechanism sublayers and a feedforward network sublayer, and the decoder adds residual connections and normalization operations after each sublayer.
[0073] It should be noted that the multi-head attention mechanism sublayer is implemented by a multi-head self-attention mechanism, which is the core module of the encoder and is responsible for learning the importance of each position from the input features and combining the information of all positions to obtain a global context representation, thereby improving the performance and generalization ability of the model. Specifically, the multi-head self-attention mechanism is implemented in that the input features pass through three different linear transformation layers to generate "query", "key" and "value" vectors respectively, and the linear transformation is implemented by multiplying the weight matrix. Further, the generated query, key and value vectors are divided into multiple smaller parts, namely "heads", so that the model processes multiple different representation subspaces in parallel, thereby capturing the diversity information in the input data. Further, for each head, the dot product of the query and key vectors is calculated respectively to obtain an attention score, which reflects the relative importance of different positions in the sequence of input data. In order to make the numerical stability of the attention score, the attention score is divided by a scaling factor, which can be the square root of the dimension of the key vector.
[0074] Furthermore, in both the encoder and decoder, the output of each sub-layer is the result of layer normalization and residual connection, which is expressed as:
[0075] y cgy =LN c (x trans ++Sublayer(x trans )),
[0076] In the formula, LN c ( ) is the layer normalization function, Sublayer( ) represents the output of the residual connected sublayer, x trans Represents the input of the sublayer; y cgy is the feature output after layer normalization.
[0077] The feedforward network sublayer consists of two fully connected layers and an activation function, which is used to perform nonlinear transformation on the vector at each position. In this sublayer, the input vector first passes through the first fully connected layer for linear transformation, then passes through a ReLU activation function for nonlinear transformation, and finally passes through the second fully connected layer for linear transformation to improve the expressiveness and generalization ability of the model, so that the model can better handle complex input data.
[0078] In summary, the autoencoding neural network training method based on feature refinement and the customized large model provided by the present invention adopt an autoencoding neural network based on feature refinement. The encoder is responsible for mapping the input high-dimensional data to a low-dimensional feature space, and the decoder is used to reconstruct the reduced-dimensional features back to the original space to ensure the reversibility of the dimensionality reduction process; the feature adjustment module dynamically adjusts the reduced-dimensional feature space through recursive feature adaptive optimization, so that important features can be strengthened and secondary features are gradually weakened, further strengthening the model's ability to capture key features, ensuring that the model does not lose valuable data information in the process of reducing dimensionality, and improving the accuracy of feature extraction and the simplicity of the model.
[0079] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0080] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0081] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0082] In the embodiments provided by the present invention, it should be understood that the disclosed systems or devices / terminal equipment and methods can be implemented in other ways. For example, the system or device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.
[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0084] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0085] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.
Claims
1. A feature refinement-based autoencoder neural network training method for Transformer network training, characterized in that: include: Step S11: The encoder uses a multi-layer nonlinear mapping structure to map the high-dimensional data in the input training data to the initial low-dimensional feature space; Step S12: After the low-dimensional features are generated, the feature adjustment module automatically generates feature weights according to the feature importance in the current feature space; wherein the feature adjustment module assigns the same initial weight to all low-dimensional features during initialization; Step S13: The feature adjustment module recursively optimizes the initially generated low-dimensional features. In each round of iteration, the feature adjustment module adjusts the weights of each feature according to the performance of the previous round of features, gradually enhances the features with important influence, and gradually weakens the redundant or noise features; Step S14: The decoder remaps the low-dimensional features back to the high-dimensional space, and then calculates the loss function to determine whether the preset stop iteration condition is met. If the result is no, the encoder maps the high-order data after the previous round of iteration to the low-dimensional feature space again for the next round of training; if the result is yes, the training ends.
2. The method for training an autoencoder neural network based on feature refinement according to claim 1, characterized in that: In step S11, the data input to the autoencoder neural network is assumed to be X r , the encoder maps the high-dimensional data to the initial low-dimensional feature space, which is expressed as: Z r =Sig enc (W r X r +b r ), In the formula, Z r represents the initial low-dimensional features, W r is the weight matrix of the encoder, b r is the bias vector of the encoder, Sig enc () is the multi-layer Sigmoid activation function of the encoder.
3. The method for training an autoencoder neural network based on feature refinement according to claim 2, characterized in that: In step S12, the calculation method of the initial weight matrix is expressed as: A r =diag(a r ), In the formula, A r is the initial weight matrix; α r is the feature weight vector, α r Each element α in r,i The same value is used at the time of initialization to indicate that all features have the same importance in the initial stage; diag() is a function for extracting the diagonal elements of the matrix; Among them, the adjusted features can be expressed as: From ′ r =A r From r , In the formula, Z ′ r is the feature representation after feature weight adjustment.
4. The method for training an autoencoder neural network based on feature refinement according to claim 3, characterized in that: In step S13, it is assumed that in the t-th iteration, the weight update rule is as follows: In the formula, represents the feature weight of the t+1th iteration, represents the feature weight of the tth iteration, η r is the learning rate, L r () is the loss function, Y r is the label data, Represents the gradient of the loss function with respect to the feature weights.
5. The method for training an autoencoder neural network based on feature refinement according to claim 4, characterized in that: In the step S13, the loss function is a reconstruction error loss function.
6. The method for training an autoencoder neural network based on feature refinement according to claim 4, characterized in that: In step S13, n r Set to 0.
01.
7. The method for training an autoencoder neural network based on feature refinement according to claim 4, characterized in that: In step S14, the reconstruction process of the decoder is expressed as: X r ′ =Sig dec (W r ′ Z r ′ +b r ′ ) Where, X r ′ is the reconstructed high-dimensional data, W r ′ is the weight matrix of the decoder, b r ′ is the bias vector of the decoder, Sig dec () is the multi-layer Sigmoid activation function of the decoder.
8. A customized large model, characterized in that: The customized large model is a model based on the Transformer network, including an encoder, a decoder and a feature adjustment module; The encoder is used to map high-dimensional data in the training data to a low-dimensional feature space; The feature adjustment module is used to dynamically adjust the feature space after dimensionality reduction through recursive feature adaptive optimization, so that important features can be strengthened and secondary features are gradually weakened; the decoder is used to reconstruct the features after dimensionality reduction back to the original space to ensure the reversibility of the dimensionality reduction process.
9. The customized large model according to claim 8, characterized in that: The customized large model is a natural language large model in the field of industrial equipment, and the training data used in its training includes: Equipment operation manuals and technical documents: record the use, maintenance and technical parameters of the equipment in detail, to help the customized large model understand the equipment operation specifications, extract technical points, and realize automatic document parsing and troubleshooting suggestions; Maintenance records and repair reports: contain equipment failure descriptions, repair steps, and spare parts used to extract common failure modes and predict maintenance needs; Fault log and alarm information: Record abnormalities in equipment operation in short text or structured data for fault mode identification and alarm optimization; Industry standards and regulatory documents: Include compliance requirements for equipment operation, which are used to help companies ensure that equipment operations comply with regulations by parsing regulatory documents.