Data preprocessing optimization method based on large star model
By introducing programmable logic controller PLC, knowledge distillation and clustering algorithms into the Star Model, and combining with the DE-CO model for data preprocessing and optimization, the efficiency and accuracy problems of distillation models in the existing technology are solved in the process of high-dimensional sparse data, and more efficient and accurate data processing and model applications are achieved.
Patent Information
- Application Number
- CN202411755718.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-11
AI Technical Summary
When using distillation models to perform complex data structure operations, the prior art has the problem of long operation time and inaccurate results. Especially when processing high-dimensional sparse data, the model is difficult to effectively apply, and the operation results lack marking, which limits its application in specific fields.
The data preprocessing optimization method based on the Star Big Model is adopted, and the Star Big Model operation is controlled through a programmable logic controller PLC and database storage and marking are performed. The data matrix is split into sub-matrix and intelligently sorted. Different types of migration are used for knowledge distillation, clustering algorithms are used for precise classification, and two-step spelling correction is performed through the DE-CO model to optimize the data processing process.
Through this method, the difficulty of obtaining multiple sub-matrix result matrices by training a large model multiple times is overcome, which reduces the high threshold for large models to allocate computing resources, improves the efficiency and accuracy of data processing, and enhances the application ability of the model in high-dimensional sparse data processing.
Smart Images
Figure CN119807790A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data elements and processing technology, and specifically relates to a method for preprocessing and optimizing data based on a large star model. Background Art
[0002] As the complexity of deep models increases, the model size increases and the speed of online reasoning gradually decreases, making it difficult to meet the needs of high QPS scenarios. Knowledge distillation is one of the optimization methods to solve such problems. Its essence is model compression and knowledge transfer.
[0003] At present, when the distillation model uses Teacher to assist Student in training complex data structure operations, the operation time is long and the results are inaccurate. There are few methods to use large models to train distillation model business data from a data-based thinking perspective. Large models have promoted the migration of distillation model business data, but some models may not be able to effectively process high-dimensional sparse data, which limits their application in certain specific fields, and the operation results are not marked, which has certain limitations on the next step of application processing. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method for preprocessing and optimizing star large model data in view of the deficiencies of the above-mentioned prior art.
[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0006] A method for preprocessing and optimizing data based on a large star model, comprising:
[0007] The programmable logic controller (PLC) is used to control the calculation of the large star model and perform database storage and marking;
[0008] The Xingchen Big Model operates on the input data to obtain a matrix, splits it into multiple sub-matrices and performs intelligent sorting;
[0009] Different types of migration of sorted data through knowledge distillation;
[0010] Clustering algorithms are used to accurately classify different types of migrated data;
[0011] The two-step spelling correction DE-CO model corrects possible hidden misspelled words.
[0012] To optimize the above technical solutions, the specific measures taken also include:
[0013] The above-mentioned knowledge distillation architecture adopts the Teacher-Student mode, taking the complex model as the Teacher and the simple model to be learned as the Student. The Teacher is used to assist the Student in training, thereby enhancing the accuracy and generalization ability of the Student weak model.
[0014] The different types of migration mentioned above include relationship-based migration, feature-based migration, and output-based migration.
[0015] The clustering steps of the above clustering algorithm are as follows:
[0016] (1) Randomly select a center u among the data points i ;
[0017] (2) For each data point x that has not yet been selected, calculate
[0018] (3) Based on Randomly select a new data point as the new center using a weighted probability distribution;
[0019] (4) Repeat steps (2) and (3) until K centers are selected;
[0020] (5) Based on the selected centers, continue to use standard K-means clustering.
[0021] In the above clustering algorithms, the complex model is used as the Teacher indicator as the center in the three groups of data classification: relationship-based migration, feature-based migration, and output-based migration.
[0022] The above DE-CO model includes a misspelled word recognizer and a misspelled word corrector;
[0023] The misspelled word identifier is used to identify misspelled words in adversarial samples and provide these misspelled words to the misspelled word corrector for correction, while ensuring that the correctly spelled words in the adversarial samples are not modified;
[0024] The misspelled word corrector is used to correct the misspelled word according to the spelling information and context information of the misspelled word.
[0025] The application process of the above DE-CO model includes:
[0026] The results of the large model operation are subjected to the first round of spelling calibration, first through the combination of misspelled word recognition and misspelled word corrector recognition; when the programmable logic controller (PLC) controls the storage of the large model, the misspelled words are marked and then stored; the PLC puts the stored sub-matrices into the small model one by one for operation.
[0027] The present invention has the following beneficial effects:
[0028] The present invention is based on the definition and architecture of knowledge distillation, starting from the two important components of knowledge distillation type and training method, and on the basis of comprehensive consideration of resources and model effects, proposes a suitable model training method and optimization technology: a programmable logic controller (PLC) is used to control the Xingchen large model operation and perform database storage and marking; the Xingchen large model operates on the input data to obtain a matrix, splits it into multiple sub-matrices and performs intelligent sorting; different types of migration are performed on the sorted data through knowledge distillation; a clustering algorithm is used to accurately classify data of different types of migration; a two-step spelling correction DE-CO model corrects possible hidden misspelled words, overcomes the difficulty of training the large model multiple times to obtain multiple sub-matrices, and reduces the threshold of the large model for high configuration of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A compression framework for knowledge distillation;
[0030] Figure 2 is the type of knowledge distillation;
[0031] Figure 3 It is a dynamic clustering box diagram;
[0032] Figure 4 The training method for the distillation model;
[0033] Figure 5 It is a two-step spelling correction model;
[0034] Figure 6 To defend against adversarial sample algorithms through DE-CO model
[0035] Figure 7 is a flow chart of the method of the present invention;
[0036] Figure 8 The schematic diagram of the method of the present invention. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0038] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.
[0039] The present invention provides a method for preprocessing and optimizing star model data. Figure 7-Figure 8 As shown, including:
[0040] The programmable logic controller (PLC) is used to control the calculation of the large star model and perform database storage and marking;
[0041] The Xingchen Big Model operates on the input data to obtain a matrix, splits it into multiple sub-matrices and performs intelligent sorting;
[0042] Different types of migration of sorted data through knowledge distillation;
[0043] Clustering algorithms are used to accurately classify different types of migrated data;
[0044] The two-step spelling correction DE-CO model corrects possible hidden misspelled words.
[0045] In the embodiment, the idea of knowledge distillation comes from transfer learning, which uses the output of a pre-trained teacher model as supervision information to train another student model. The teacher model is often a network with good performance, complexity and good generalization ability, which is used to guide another student model with fewer parameters and simpler, so that the student model has the same performance as the teacher model.
[0046] Hinton introduced the knowledge distillation compression framework in a paper published in 2015, such as Figure 1 shown.
[0047] Here, the student network training has two objectives: one is the original objective function, also called the hard target (HardTarget), which is the cross entropy between the category probability output of the student model and the true label; the other is the soft target (SoftTarget), which is the cross entropy between the category probability output of the student model and the category probability output of the teacher model. The soft target is obtained by adjusting the softmax output of the original model, adding a temperature constant T to control the smoothness of the predicted probability. The output of the adjusted Softmax function is The implementation process of this method is as follows: First, a larger T value and hard target are used to train the teacher model. A larger T value will generate a softer probability distribution. This method generates soft labels, and then a simple student model is trained using the soft target. The objective function of the student model consists of a weighted average of hard and soft targets, where the weight of the soft target should be larger. In this way, even when there is little data, the trained student model can have comparable performance to the teacher model, but the complexity and computational complexity are greatly reduced.
[0048] The architecture of knowledge distillation usually adopts the Teacher-Student model. The complex model is used as the Teacher, and the simple model to be learned is used as the Student. The Teacher is used to assist the Student training to enhance the accuracy and generalization ability of the Student weak model. Knowledge distillation includes two important components: one is the type of knowledge distillation, that is, which part of the knowledge is transferred; the other is the training method of knowledge distillation, that is, based on the comprehensive consideration of resources and model effects, the appropriate model training method and optimization technology are adopted. Clustering algorithms are used to accurately classify the data of relationship-based migration, feature-based migration, and output-based migration in historical data.
[0049] like Figure 2 As shown in the figure, from the perspective of the type of knowledge distillation, it is divided into relationship-based transfer, feature-based transfer, and output-based transfer.
[0050] Furthermore, clustering algorithms are used to accurately classify the relationship-based migration, feature-based migration, and output-based migration data in historical data. The K-means++ algorithm is divided into the following steps:
[0051] (1) Randomly select a center u among the data points i ;
[0052] (2) For each data point x that has not yet been selected, calculate That is, the distance between z and the closest center that has been selected;
[0053] The three groups of data classification, namely relationship-based migration, feature-based migration and output-based migration, take the complex model as the center of the Teacher indicator.
[0054] (3) Use the weighted probability distribution to randomly select a new data point as the new center, where the probability of selecting point z is is proportional to;
[0055] (4) Repeat (2) and (3) until K centers are selected (i.e., j = K);
[0056] (5) Now that the initial centers have been selected, continue using standard K-means clustering.
[0057] It is easy to see that the center obtained in this way can minimize the sum of squared deviations under the classification conditions at the time of iteration.
[0058] (4) If This holds true for all j = 1, 2, ..., K, that is, the classification and center position of any point have not changed after the iteration correction, then the algorithm is said to have converged and can be terminated. Otherwise, go to (2) to continue correcting the sample division and the positions of the K center points.
[0059] The corresponding dynamic clustering process is as follows Figure 3 shown.
[0060] In the embodiment, the traditional rough sorting and fine sorting are arranged in descending order according to the scores, and then the top K are selected and sent to the fine sorting for comparison. The rough sorting positioning and technical route selection are as follows:
[0061] As a link between recall and fine ranking, coarse ranking pays more attention to sorting than recall, that is, after sorting in descending order according to the score, the TopK items are selected and sent to fine ranking. Since the TopM of the fine ranking result is much smaller than the TopK, in order to enhance the screening efficiency (the recommended items that are more suitable for fine ranking can reach fine ranking through the pre-processing link), coarse ranking needs to maintain a certain degree of consistency with the fine ranking target. From a phased perspective, coarse ranking can be considered as a special "recall" before fine ranking. Compared with the sample space of fine ranking, which is the exposed items, the sample space of coarse ranking is the result after recall or the whole library. Compared with fine ranking, coarse ranking pays more attention to performance. Since the structure of the fine ranking model is usually very complex, when the number of item libraries is large, it is difficult to return the scoring results in a short time in the face of the collection of items produced by multi-way recall. Therefore, the starting point of coarse ranking is a "simplified version of fine ranking", that is, the migration or compression of fine ranking, which means that it is necessary to meet the performance requirements as much as possible on the basis of meeting the sorting accuracy.
[0062] Figure 4 The technical route of coarse sorting is demonstrated. The evolution of the coarse sorting model is basically optimized in the direction of "reducing the gap in estimation capabilities between coarse sorting and fine sorting" and "enhancing link target consistency".
[0063] Compared with the traditional rough sorting and fine sorting, which selects the top K and sends them to the fine sorting after sorting in descending order according to the photo scores, the present invention innovatively adopts the large model method for sorting.
[0064] First, the large model operates on the input data to obtain a matrix and splits it into multiple sub-matrices.
[0065] Secondly, a programmable logic controller (PLC) is used to control the large model operation result matrix and store it in the database.
[0066] Then, a two-step spelling correction (Detector-Corrector, DE-CO) model is constructed to correct possible hidden misspelled words and perform operations in the context data to improve the accuracy of the small model calculation.
[0067] Through the control of the programmable logic controller (PLC), the difficulty of training a large model multiple times to obtain multiple sub-matrix result matrices is overcome, and the threshold for large models to configure high computing resources is lowered.
[0068] The keyword extraction model, namely the two-step spelling correction (Detector-Corrector, DE-CO) model structure is as follows Figure 5 shown.
[0069] The results of the large model operation are subjected to the first round of spelling calibration, first through the combination of misspelled word recognition and misspelled word corrector recognition.
[0070] When the programmable logic controller (PLC) controls the storage of the large model, the misspelled words are marked and then stored, thus overcoming the drawback of not marking the retrieval results of the large model.
[0071] The PLC puts the stored sub-matrices into the small model one by one for calculation. Through the control of the programmable logic controller (PLC), the difficulty of training the large model multiple times to obtain multiple sub-matrices is overcome, which reduces the threshold for large models to configure high computing resources.
[0072] The DE-CO model consists of a misspelled word recognizer and a misspelled word corrector. The function of the misspelled word recognizer is to recognize misspelled words in adversarial samples and provide these misspelled words to the misspelled word corrector for correction, while ensuring that the correctly spelled words in the adversarial samples are not modified. The function of the misspelled word corrector is to correct the misspelled words based on their spelling information and context information. Figure 6 The algorithm shown describes the workflow of the DE-CO model.
[0073] The glossary of terms in this article is explained as follows:
[0074] Data elements refer to data resources that exist in electronic form, participate in production and operation activities through computing, and play an important role. In the digital economy, the role of data elements is comparable to traditional production factors (such as labor, capital, and land). Data elements are the core engine driving the development of the digital economy, an important support for enabling the digital transformation and intelligent upgrading of industries, and a national basic strategic resource.
[0075] Knowledge distillation is a machine learning method mainly used for model compression and knowledge transfer. It improves the performance of the student model by using a larger model (teacher model) to train a smaller model (student model) and transfers the knowledge of the teacher model (such as weights, predictions, etc.) to the student model while reducing the number of parameters and computational complexity.
[0076] Programmable Logic Controller (PLC) is a digital computing electronic system designed for use in industrial environments. It uses a programmable memory to store instructions for performing logical operations, sequential control, timing, counting and arithmetic operations, and controls various types of mechanical equipment or production processes through digital or analog input and output.
[0077] Large models are widely used in natural language processing, computer vision, speech recognition and other fields. These models usually have millions to billions of parameters and can be trained with massive data and powerful computing power to achieve efficient processing of various tasks. The design and training of large models are aimed at providing more powerful and accurate model performance to cope with more complex and larger data sets or tasks. These models are usually able to learn more subtle patterns and regularities and have stronger generalization and expression capabilities.
[0078] China Telecom Artificial Intelligence Technology Co., Ltd. and China Telecom (601728) Artificial Intelligence Research Institute (TeleAI) released a product called Xingchen Big Model Software Factory (hereinafter referred to as Xingchen Software Factory), and said that this is the first breakthrough AI development tool in China.
[0079] Small models usually refer to machine learning or deep learning models with fewer parameters and shallower layers. They are lightweight, efficient, easy to deploy, and can run efficiently in resource-limited environments such as mobile devices or embedded systems.
[0080] K-means++ is an algorithm that selects initial values (or "seeds") for the K-means clustering algorithm. The location of the K initial centroids has a great impact on the final clustering results and running time, so it is necessary to select K appropriate centroids.
[0081] The basic idea of K-means algorithm is still to minimize the K-means sum of squares, but in its specific implementation, it adopts the method of successive iterative correction.
[0082] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0083] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A method for preprocessing and optimizing star model data, characterized in that: include: The programmable logic controller (PLC) is used to control the calculation of the large star model and perform database storage and marking; The Xingchen Big Model operates on the input data to obtain a matrix, splits it into multiple sub-matrices and performs intelligent sorting; Different types of migration of sorted data through knowledge distillation; Clustering algorithms are used to accurately classify different types of migrated data; The two-step spelling correction DE-CO model corrects possible hidden misspelled words.
2. The method for preprocessing and optimizing star model data according to claim 1, characterized in that: The architecture of the knowledge distillation adopts the Teacher-Student mode, taking the complex model as the Teacher and the simple model to be learned as the Student. The Teacher is used to assist the Student in training, thereby enhancing the accuracy and generalization ability of the Student weak model.
3. The method for preprocessing and optimizing star model data according to claim 1, characterized in that: The different types of migration include relationship-based migration, feature-based migration, and output-based migration.
4. The method for preprocessing and optimizing star model data according to claim 1, characterized in that: The clustering steps of the clustering algorithm are as follows: (1) Randomly select a center u among the data points i ; (2) For each data point x that has not yet been selected, calculate (3) Based on Randomly select a new data point as the new center using a weighted probability distribution; (4) Repeat steps (2) and (3) until K centers are selected; (5) Based on the selected centers, continue to use standard K-means clustering.
5. The method for preprocessing and optimizing star model data according to claim 1, characterized in that: In the clustering algorithm, the complex model is used as the Teacher indicator as the center in the three groups of data classification, namely, relationship-based migration, feature-based migration and output-based migration.
6. The method for preprocessing and optimizing star model data according to claim 1, characterized in that: The DE-CO model includes a misspelled word recognizer and a misspelled word corrector; The misspelled word identifier is used to identify misspelled words in adversarial samples and provide these misspelled words to the misspelled word corrector for correction, while ensuring that the correctly spelled words in the adversarial samples are not modified; The misspelled word corrector is used to correct the misspelled word according to the spelling information and context information of the misspelled word.
7. The method for preprocessing and optimizing star model data according to claim 6, characterized in that: The application process of the DE-CO model includes: The results of the large model operation are subjected to the first round of spelling calibration, first through the combination of misspelled word recognition and misspelled word corrector recognition; when the programmable logic controller (PLC) controls the storage of the large model, the misspelled words are marked and then stored; the PLC puts the stored sub-matrices into the small model one by one for operation.