Model training methods and text processing methods

Abstract

The embodiments of this specification provide a model training method and a text processing method, wherein the model training method includes: obtaining at least two reference models, and obtaining at least two target data processing networks based on each reference model, wherein the target data processing network includes at least one full data processing network and at least one compressed data processing network; generating a data processing model according to each target data processing network, wherein the data processing model includes at least one data encoding layer, and the data encoding layer includes at least one full data processing network and at least one compressed data processing network; obtaining target sample data, and training the data processing model based on the target sample data until the data processing model reaches the model training stop condition. By constructing the target data processing network through the full and compressed data processing networks, the resource consumption of large model training can be effectively reduced, thereby improving the efficiency of data processing while ensuring the accuracy of data processing.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a model training method. Background Art

[0002] With the continuous development of large model technology, the requirements for model performance and scale are increasing, which has led to a significant increase in training costs. In order to make the training process converge smoothly and avoid catastrophic forgetting of other data, the training set must be continuously expanded, resulting in an increase in resource consumption and electricity costs.

[0003] Currently, the cost of model training is reduced by controlling the training model through a gating mechanism, solving the resource consumption problem caused by large-scale model training. However, when dealing with simple problems, this technology uses a larger model, which slows down the response speed and fails to balance speed and performance. Therefore, in order to overcome these shortcomings, a model training method is needed. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a model training method, a text processing method, and a text processing method applied to a cloud device. One or more embodiments of this specification also involve a training platform, a model training device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a model training method is provided, including:

[0006] Acquire at least two reference models, and acquire at least two target data processing networks based on the reference models, wherein the target data processing networks include at least one full data processing network and at least one compressed data processing network;

[0007] Generating a data processing model according to each target data processing network, wherein the data processing model includes at least one data encoding layer, and the data encoding layer includes at least one full data processing network and at least one compressed data processing network;

[0008] Target sample data is acquired, and the data processing model is trained based on the target sample data until a training stop condition of the data processing model is reached, wherein the target sample data is data used to train the data processing model.

[0009] According to a second aspect of the embodiments of this specification, a text processing method is provided, including:

[0010] Get the text to be processed;

[0011] The text to be processed is input into a data processing model to obtain a text processing result output by the data processing model, wherein the data processing model is trained by the above-mentioned model training method.

[0012] According to a third aspect of the embodiments of this specification, a text processing method is provided, which is applied to a cloud device, including:

[0013] A text processing request sent by a receiving terminal device, wherein the text processing request includes a text to be processed;

[0014] Inputting the text to be processed into a data processing model to obtain a text processing result output by the data processing model, wherein the data processing model is trained by the above-mentioned model training method;

[0015] The text processing result is sent to the terminal device.

[0016] According to a fourth aspect of an embodiment of this specification, there is provided a training platform, including a request interface and a response unit;

[0017] The request interface is used to receive a task generation request sent by a terminal device;

[0018] The response unit, based on the task generation request, obtains a data processing model, wherein the data processing model is trained by the above-mentioned model training method; based on the data processing model, generates task information, wherein the task information is used for the terminal device to perform the text processing task.

[0019] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0020] Memory and processor;

[0021] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned model training method, text processing method and text processing method applied to cloud devices are implemented.

[0022] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned model training method, text processing method, and text processing method applied to a cloud device.

[0023] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned model training method, text processing method, and text processing method applied to a cloud device.

[0024] By applying the solution of the embodiments of this specification, by obtaining multiple reference models and building a target data processing network based on them, the model structure can be flexibly adjusted to meet the needs of different tasks and application scenarios. The combination of the full data processing network and the compressed data processing network not only optimizes the parameter quantity and computing efficiency of the model, but also enables the model to automatically select a more suitable network path according to specific needs at different training stages. This method improves the utilization of computing resources while ensuring the processing capacity of high-precision tasks, effectively reducing resource consumption during training and reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of a model training method provided by an embodiment of this specification;

[0026] Figure 2 It is a structural diagram of a data processing model provided by an embodiment of this specification;

[0027] Figure 3 is a flowchart of a text processing method provided by an embodiment of this specification;

[0028] Figure 4 is a flowchart of a text processing method applied to a cloud device provided by an embodiment of this specification;

[0029] Figure 5 is an architecture diagram of a text processing system provided by an embodiment of this specification;

[0030] Figure 6 is a process flow chart of an automatic question-answering model training method provided by an embodiment of this specification;

[0031] Figure 7 It is a structural schematic diagram of a model training device provided by an embodiment of this specification;

[0032] Figure 8 It is a structural diagram of a cloud training platform provided by an embodiment of this specification;

[0033] Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0034] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0035] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0037] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0039] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0040] First, the terms involved in one or more embodiments of this specification are explained.

[0041] Mixture of Experts (MoE): is a machine learning method that improves task performance by combining multiple sub-models (expert models). In this model, different expert models process different input features, and the degree of participation of each expert model is controlled by a gating mechanism. By dynamically selecting appropriate expert models for calculation, MoE can more efficiently utilize the expertise of different experts. Its structure is flexible and can automatically adjust the participating experts according to the different requirements of the task, thereby improving the expressiveness and performance of the model.

[0042] In this specification, a model training method, a text processing method, and a text processing method applied to a cloud device are provided. One or more embodiments of this specification also involve a training platform, a model training device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0043] MoE (Mixture of Experts) is a common cost control solution after the current large-scale model is scaled and scaled. It is a very important part of the future long-term strategy of large models. This technology determines whether we can control the cost of electricity, energy, etc. after providing services on a large scale and in a normalized manner in the future.

[0044] According to a recent report, an AI model responds to about 200 million requests every day, consuming more than 500,000 kWh of electricity, equivalent to the daily electricity consumption of 17,000 households. This huge power consumption would require significantly more electricity without the MoE framework. The MoE framework not only significantly reduces costs, but also ensures that high performance requirements are met. In actual applications, for 80% of simple problems, small models can quickly provide accurate answers. However, unified large models can solve more complex problems, although this may result in slower response times.

[0045] See also Figure 1 , Figure 1 A flowchart of a model training method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0046] Step 102: Acquire at least two reference models, and acquire at least two target data processing networks based on the reference models, wherein the target data processing networks include at least one full data processing network and at least one compressed data processing network.

[0047] In practical applications, the reference model is the initial model used for model construction and training. The target data processing network is a structural network built based on the reference model and used for data processing tasks. The full data processing network is a structure that utilizes the entire network resources when processing data. The compressed data processing network is a network that reduces the amount of computation and model parameters by optimizing the network structure in specific scenarios.

[0048] Exemplarily, the reference model can be understood as the basic model provided for each expert module in the process of building the data processing model. Each expert module has its own unique functions and processing methods. In the framework of the entire data processing model, multiple expert models are combined to complete the task. The reference model here refers to the basic structure used to initialize the expert module. They may have been trained in the pre-training stage and used as the basis for building a complex data processing model architecture. For example, an expert module may be based on a simpler language model, while another expert module may rely on a more complex deep neural network model. Different reference models can provide customized solutions for different input data.

[0049] The data processing model can be understood as a model built through different data processing networks in a specific task, and can be understood as a deep learning model built based on the MoE framework. The data processing model selects appropriate experts to process tasks through different expert modules and routing mechanisms, thereby improving the adaptability and efficiency of the model in multiple tasks. The data processing model is essentially a specific manifestation of the MoE model. Through carefully designed expert modules and selection mechanisms, it can effectively and dynamically select more suitable processing paths for different data tasks. The design of the data processing model is crucial to the accuracy and computational efficiency of processing tasks. For example, in speech recognition tasks, the data processing model can select different experts for processing according to the complexity of the task. Simple tasks are processed by smaller experts, and complex tasks are processed by large experts, avoiding redundant calculations and improving the efficiency of overall task processing.

[0050] The target data processing network can be understood as an expert module generated based on the reference model for processing actual data. In the data processing model, the target data processing network is usually composed of multiple expert modules, each of which responds differently according to the different characteristics of the input data, thereby effectively improving the processing capacity of the entire system. Depending on the task requirements, these target data processing networks may be full data processing networks or compressed data processing networks, which are determined by the complexity of the task and the demand for computing resources.

[0051] The full data processing network can be understood as a network that uses a complete network structure to process data. It does not perform any compression or simplification, and makes full use of all network parameters and computing resources to pursue high-precision results. In some tasks that require processing high-precision and large-scale data, the use of a full data processing network can ensure the quality of the output results. For example, when performing complex natural language understanding, a full data processing network may be used to ensure the performance of the model.

[0052] A compressed data processing network can be understood as a network that improves efficiency by reducing the computational complexity of the network and the number of network parameters when processing data. Compression technology can be implemented through methods such as pruning, quantization, or low-rank decomposition. For some resource-constrained devices or scenarios, the use of a compressed data processing network can significantly reduce computing and storage overhead while ensuring good performance. For example, mobile devices often use compressed data processing networks to adapt to the limitations of computing power and storage capacity, while finding a balance between processing speed and accuracy.

[0053] By combining the full data processing network and the compressed data processing network, the efficiency of computing resource utilization can be optimized without affecting task accuracy. This combination enables the system to select the most suitable network path according to needs at different tasks and data processing stages, thereby improving efficiency while reducing resource consumption.

[0054] Further, obtaining at least two target data processing networks based on each reference model includes:

[0055] Determine a target reference model, wherein the target reference model is any one of the reference models;

[0056] A reference model data processing layer is determined in the target reference model, and a target data processing network is acquired according to the reference model data processing layer.

[0057] In practical applications, the reference model data processing layer is the neural network layer in the reference model that has a greater impact on the model output results. It usually contains the main computing units in the neural network, which can deeply analyze and transform the features of the input data and generate feature representations suitable for downstream tasks. The role of the model data processing layer is to provide basic data conversion and feature extraction support for the entire model, which directly affects the accuracy and expressiveness of the model output. For example, in an image recognition model, the model data processing layer may be composed of a set of convolutional layers that extract features from the pixel information of the image and convert the image into a more structured feature map for further analysis by subsequent layers. In natural language processing tasks, the model data processing layer may contain a multi-layer self-attention mechanism to extract contextual associations and semantic information from sentences, thereby improving the accuracy of language understanding.

[0058] It should be noted that determining the model data processing layer in the reference model can be understood as finding the network layer in the model that is crucial for data feature extraction and processing. The specific method is to analyze the structural characteristics of the reference model to identify the network layer with outstanding feature extraction effect, such as identifying the deeper convolution layer as the data processing layer in the convolutional neural network; or to select the layer with higher weight by calculating the weight of each layer's influence on the model output; or to observe the gradient changes of each layer during the model training process, and select the layer with larger gradient to determine the data processing layer, thereby ensuring the effective transmission of feature information. This manual does not impose any restrictions on this.

[0059] By determining the reference model data processing layer in the target reference model and determining the target data processing network based on the reference model data processing layer, the representative layer in the reference model can be effectively extracted. Compared with directly using the reference model to obtain the target data processing network, the computing and storage overhead can be significantly reduced while ensuring good performance.

[0060] Further, obtaining a target data processing network according to the reference model data processing layer includes:

[0061] Acquire initial parameter information corresponding to the reference model data processing layer;

[0062] According to the initial parameter information, obtaining parameter characteristic information corresponding to the reference model data processing layer;

[0063] A compressed data processing network is generated based on the parameter characteristic information.

[0064] In practical applications, the initial parameter information is the original parameter used to describe the model weights in the data processing layer of the reference model; the parameter feature information is the key statistical or structured information extracted based on the initial parameter information, reflecting the characteristics and importance of the parameters.

[0065] Initial parameter information can be understood as the original weight configuration or parameter setting of the data processing layer in the reference model. These parameters are obtained by the reference model during the training phase and are used to reflect the basic capabilities of the model in the data processing process. Parameter feature information can be understood as extracting feature information that reflects the importance, distribution characteristics, or value range of each parameter based on the initial parameter information. By extracting features from the initial parameter information, parameter information or a combination of specific parameter information that contributes more to model performance can be identified.

[0066] It should be noted that obtaining the parameter feature information corresponding to the reference model data processing layer according to the initial parameter information can be understood as extracting and analyzing the key feature data in the initial parameters. The specific method can be to calculate the mean and variance of the parameters, analyze the parameter distribution characteristics, and use them to determine the contribution of different parameters; it can also calculate the initial parameters only for the characteristic information of scaling and output to identify important features and optimize the model structure; it can also calculate the gradient changes of each parameter through gradient analysis to determine the parameters that have a greater impact on the output results and focus on them. This manual does not impose any restrictions on this.

[0067] By generating a compressed data processing network based on parameter feature information generated based on initial parameter information, the amount of computation and storage requirements can be significantly reduced while ensuring that the key features of the model are not lost, thereby improving the operating efficiency of the model. Parameter feature information helps identify redundant parts that contribute less to model performance, making the compression process more accurate and avoiding unnecessary performance loss. The generated compressed data processing network can achieve lower resource consumption and increase processing speed while meeting task requirements, providing the model with more advantageous performance in resource-limited environments.

[0068] Further, obtaining parameter characteristic information corresponding to the reference model data processing layer according to the initial parameter information includes:

[0069] Based on the initial parameter information, obtaining parameter scaling characteristic information and parameter output characteristic information;

[0070] Parameter characteristic information is acquired according to the parameter scaling characteristic information and the parameter output characteristic information.

[0071] In practical applications, parameter scaling feature information is information extracted from the initial parameters to describe the proportional changes of parameters in different dimensions; parameter output feature information is feature information that indicates the degree of influence of each parameter in the model output layer on the output result. Parameter scaling feature information can be understood as feature data for proportional adjustment of each parameter in the model in different dimensions. Parameter output feature information can be understood as a measure of the influence of each parameter in the model on the final output result.

[0072] In one embodiment provided in this specification, the method of obtaining parameter scaling characteristic information and parameter output characteristic information is as shown in Formula 1:

[0073] ...Formula 1

[0074] Among them, A is the initial parameter information, U is the parameter output feature information, Σ is the scaling feature information, and V is the input feature information that defines the transformation direction of the input data.

[0075] It should be noted that the main purpose of obtaining parameter feature information based on parameter scaling feature information and parameter output feature information is to more comprehensively understand and optimize model parameters, so as to maintain performance during model compression and optimization. Parameter scaling feature information and parameter output feature information respectively reflect the model's ability to process data in different dimensions and the influence of each parameter on the output results. The combination of this information can provide a deeper parameter analysis and help identify which parameters are critical in maintaining model performance and which parameters may be redundant.

[0076] By integrating these feature information, the core parameters can be effectively screened out during model compression, reducing unnecessary computational and storage burdens. For example, higher parameter scaling feature information and parameter output feature information often indicate the importance of the parameters in a specific task. Retaining or optimizing these parameters can ensure that the compressed model still maintains accuracy and computational efficiency. Therefore, by obtaining this feature information, the model can effectively improve computational efficiency in a resource-constrained environment while ensuring that performance is not significantly affected, which is of significant value for deployment and maintenance in practical applications.

[0077] Further, obtaining a target data processing network according to the reference model data processing layer includes:

[0078] Acquire initial parameter information corresponding to the reference model data processing layer;

[0079] The initial parameter information is determined to be a full data processing network.

[0080] In practical applications, the target data processing network is obtained according to the reference model data processing layer, and the initial parameter information corresponding to the reference model data processing layer is obtained, ensuring that the generated target data processing network can retain the basis of the reference model performance. Since the initial parameter information usually contains information such as weights and biases obtained during the model training process, it determines the performance of the model on different tasks. Therefore, by obtaining the initial parameter information, it can be ensured that the target data processing network can inherit the characteristics of the reference model when processing specific data, and lay the foundation for further optimization.

[0081] Exemplarily, directly determining the initial parameter information as the full data processing network can retain the data processing capabilities of the reference model to ensure the processing capabilities and accuracy of the subsequently generated data processing model in complex tasks. In this way, the target data processing network in the data processing model can better cope with complex data features, thereby improving the adaptability and robustness of the model. This process not only helps to ensure the efficiency of the data processing model, but also provides more flexible applications in different environments, optimizing processing speed and performance, especially in tasks with large-scale data processing and high precision requirements.

[0082] Step 104: Generate a data processing model according to each target data processing network, wherein the data processing model includes at least one data encoding layer, and the data encoding layer includes at least one full data processing network and at least one compressed data processing network.

[0083] In practical applications, the data coding layer is the network layer that actually processes data, and is mainly used to perform deep processing on input data. The data coding layer contains at least one full data processing network and one compressed data processing network. The full data processing network is responsible for processing the original data and retaining rich information details; the compressed data processing network reduces the redundant parts and optimizes the use of computing resources by appropriately processing the data.

[0084] The data encoding layer can be understood as a network layer dedicated to deep data processing, which aims to improve model performance through different levels of processing. The full data processing network is suitable for tasks that require maintaining all details to ensure information integrity; while the compressed data processing network reduces computing resource consumption and accelerates the data processing process by optimizing data processing methods. In practical applications, such as natural language processing tasks, the full data processing network retains all semantic information to help the model better understand the text, while the compressed data processing network can effectively compress data, improve processing efficiency, and ensure accuracy.

[0085] Furthermore, a data processing model is generated according to each target data processing network, including:

[0086] Determining a target full data processing network and at least one target compressed data processing network in each target data processing network, wherein the target full data processing network is any one of the full data processing networks, and the target compressed data processing network is any one of the compressed data processing networks;

[0087] Acquire a target data coding layer based on the target full data processing network and each target compressed data processing network, wherein the target data coding layer is any one of the data coding layers;

[0088] A data processing model is generated according to each target data encoding layer.

[0089] In practical applications, the target data coding layer is obtained based on the target full data processing network and each target compressed data processing network. It can be understood as generating the data coding layer of the data processing model based on a full data processing network and one or more compressed data processing networks. The core purpose of this step is to ensure that the selected target full data processing network can cover a wider range of data processing capabilities and is suitable for various complex task scenarios. At the same time, a target compressed data processing network is selected from multiple compressed data processing networks to optimize the storage efficiency of data processing and the utilization of computing resources. The selection of these two networks helps to achieve the efficiency and balance of the model, ensuring that the model can maintain high performance and good computing efficiency when processing high-dimensional data.

[0090] By constructing the data coding layer, the input data can be effectively processed in multiple dimensions and angles, helping the model to better understand and extract important features from the data. The construction of the data coding layer ensures that the model has higher flexibility and adaptability, especially when faced with complex and dynamically changing input data. The steps of generating a data processing model based on each data coding layer mark the process of establishing a complete data processing model. This process not only helps to improve the performance of the model in different tasks, but also maintains the accuracy of the model while optimizing computational efficiency. By flexibly adjusting the parameters and configurations of each layer, the model can adapt to more diverse application scenarios and requirements, especially in large-scale data sets and real-time processing requirements, and can ensure efficient and stable performance.

[0091] Furthermore, obtaining a target data coding layer based on the target full data processing network and each target compressed data processing network includes:

[0092] Acquire a path selection network, wherein the path selection network is used to select at least one data processing network for data processing in this round from among the target full data processing network and each target compressed data processing network;

[0093] A target data coding layer is obtained according to the path selection network, the target full data processing network and each target compressed data processing network.

[0094] In practical applications, the path selection network is a neural network that selects data processing in the data encoding layer for data processing in this round. The path selection network can be understood as an intelligent selection mechanism that decides which data processing network to use for task processing based on specific input conditions or task requirements. For example, when processing image data, the path selection network may select several more suitable compression networks to improve the accuracy of data processing based on the complexity or other characteristics of the image. The role of the path selection network is to enhance the flexibility of the system so that the entire data processing process can be dynamically adjusted according to different data or task requirements, so as to achieve the purpose of optimizing resource allocation and improving processing efficiency.

[0095] It should be noted that the use of a path selection network to determine a data processing network for data processing in multiple compressed data processing networks can be understood as the path selection network selecting one or more suitable data processing networks for task processing based on the fit between the current data and each compressed data processing network. The specific method of the path selection network can be to calculate the similarity or matching degree between the current data and each network, for example, to select a better network using a specific similarity measurement method; the selection strategy can also be dynamically adjusted according to the characteristics of the data, that is, to identify the key attributes in the data (such as dimension, distribution, noise, etc.), and compare the processing capabilities of different networks (such as network layer depth, convolution kernel size, activation function, etc.) and the matching degree between these features, and give priority to networks with higher matching degree with data features when processing large-scale or complex data; the selection criteria of the path selection network can also be continuously optimized through a real-time feedback mechanism, so that it can more accurately select suitable networks for processing different data, etc., and this manual does not impose any restrictions on this.

[0096] refer to Figure 2 , Figure 2 A structural schematic diagram of a data processing model provided for an embodiment of the present specification, wherein the left side is the overall structure of the data processing model, the data processing model has a data embedding layer, at least one data encoding layer and a data decoding layer, the data embedding layer is connected to the first data encoding layer, each encoding layer is connected in sequence, and the last data encoding layer is connected to the data decoding layer; the right side is a further structure of the data encoding layer in the data processing model, the data encoding layer has a self-attention network, a path selection network, a full data processing network and one or more compressed data processing networks, a data fusion network, a normalization network and an activation network, and the data input to the data encoding layer is first input To the self-attention network, and then the self-attention network inputs the processed data into the full data processing network and the path selection network respectively. The path processing network determines which compressed data processing network to use for further data processing according to the input data, and the data generated by the full data processing network and the data generated by the compressed data processing network selected by the path selection network are input into the data fusion network to obtain the fused feature data, and then the fused feature data is input into the normalization network to make the above fused features more continuous, and the normalized data is input into the activation network so that the data can be input into the subsequent data encoding layer or data decoding layer for further processing.

[0097] In practical applications, the data embedding layer is a network layer that converts the original input data into an embedded representation that can be processed by the model; the data decoding layer is a network layer that generates the final output from the processed representation; the term "self-attention network" is a network layer used to focus on the internal correlation of the input data; the data fusion network is a network layer that integrates feature information from different data processing paths; the normalization network is a network layer that standardizes the data to improve the stability of the model; the activation network is a network layer that introduces nonlinear features into the processed data.

[0098] The data embedding layer can be understood as a network layer responsible for converting raw input data (such as text or images) into the embedding vector required by the model. This step enables the model to effectively process input information by compressing high-dimensional data into the form of low-dimensional vectors. For example, in natural language processing tasks, the data embedding layer converts each word into a word vector so that the semantic information of the word can be passed to subsequent network layers. The data decoding layer can be understood as a network layer used to convert the embedded representation processed by the model into an interpretable output. This layer is the final output layer of the data processing model and is responsible for generating results that meet the task requirements. For example, in a machine translation task, the data decoding layer converts the encoded language features into text in the target language to complete the translation task.

[0099] The self-attention network can be understood as a network layer used to calculate the relationship between the various parts of the data, which can capture the dependencies within the input sequence. This network extracts more useful feature information by focusing on different parts of the input data. For example, in text processing tasks, the self-attention network can identify the relationship between words in a sentence and help the model understand the context structure of the sentence. The data fusion network can be understood as a network layer that integrates feature information from different sources or paths, and is used to merge the feature data generated by the full data processing network and the compressed data processing network into a comprehensive representation. The data fusion network can help the model efficiently combine information from different feature sources during multi-task learning or multi-channel data processing to improve the comprehensiveness of data processing.

[0100] The normalization network can be understood as a network layer that standardizes the input data to adjust the data distribution to make it more suitable for model processing. This layer is usually used after feature fusion. Through the normalization operation, the impact of numerical fluctuations can be reduced, helping the model to remain stable during training and inference. For example, batch normalization is one of the common normalization methods. The activation network can be understood as a network layer that introduces nonlinear features to the data through nonlinear activation functions. This layer is usually applied after normalization. The purpose is to increase the expressive power of the model through the activation function, so that it can learn complex patterns and features, thereby improving the performance of the model when processing complex data.

[0101] By constructing the data encoding layer from a path selection network, it is possible to effectively and dynamically select a more appropriate compressed data processing network based on the characteristics of the current data, thereby optimizing the use of resources. This approach not only improves processing efficiency, but also reduces unnecessary computational overhead, making the data encoding layer more flexible under different task requirements. In addition, the path selection network can adaptively select the optimal processing path based on the characteristics of the input data, ensuring that the model maintains a high response speed while ensuring performance. This structural design helps to balance processing quality and computational cost in complex data processing scenarios, and provides more representative feature data for subsequent network layers.

[0102] Step 106: Acquire target sample data, and train the data processing model based on the target sample data until the data processing model reaches a model training stop condition, wherein the target sample data is data used to train the data processing model.

[0103] In practical applications, target sample data refers to the specific data set used to train the model, and the model training stop condition refers to the standard or condition used to determine when to stop training during the training process.

[0104] Exemplarily, the target sample data can be understood as the input data set used in the model training process, which can be part of any public data set or a data set for a target project. The model training stop condition can be understood as a standard for deciding when to end the training process. Generally, the training stop condition includes several aspects, such as whether the performance of the model on the validation set corresponding to the target sample data reaches the predetermined goal, or whether the error of the model converges to a fixed low level. It may also include a limit on the number of training times, a limit on the training time, or stop when the validation loss is no longer significantly improved. For example, when the validation error of the model no longer decreases significantly after several rounds of training, it is usually considered that the model has been trained well enough and the training can be stopped to avoid overfitting caused by overtraining. In practical applications, reasonably setting the training stop condition helps to improve the efficiency of model training and prevent unnecessary waste of computing resources.

[0105] Further, training the data processing model based on the target sample data until the data processing model reaches a model training stop condition includes:

[0106] Based on the target sample data, the target parameters corresponding to the data processing model and / or the path selection parameters corresponding to the path selection network in each data encoding layer are adjusted until the data processing model reaches the model training stop condition.

[0107] In practical applications, the target parameters are the parameters of each neural network layer in the data processing model, which are used to optimize the model performance, and the path selection parameters are the parameters in the path selection network, which are used to control the selection mechanism of different data processing networks.

[0108] Exemplarily, target parameters can be understood as key parameters that directly affect model performance and output results during model training. These parameters are usually weights and bias items adjusted by the model through optimization algorithms during training, aiming to enable the model to achieve the expected results when processing data. For example, in a neural network, the target parameters include the weight matrix and bias items of each layer. The adjustment of these parameters directly determines the accuracy and generalization ability of the model. By optimizing the target parameters, the model can perform better on a given task and adapt to different data characteristics.

[0109] Path selection parameters can be understood as control parameters used to determine the direction of data flow in the path selection network. These parameters are used to dynamically select the data processing network used for data processing based on the characteristics of the input data to ensure that the data is routed to a more appropriate compressed data processing network or full data processing network. The optimization of path selection parameters can effectively improve the processing efficiency of the model, allowing it to automatically select the best processing path under complex tasks or different input conditions. For example, in the environment of multi-task learning, path selection parameters can help the model decide which data processing network to call based on task requirements, thereby achieving a balance between computational efficiency and processing accuracy.

[0110] It should be noted that when training the data processing model, you can choose to adjust only the target parameters or the path selection parameters, or you can choose to adjust both the target parameters and the path selection parameters, and this specification does not impose any restrictions on this. By choosing to adjust only the target parameters or the path selection parameters, or choosing to adjust both the target parameters and the path selection parameters, you can flexibly optimize different aspects of the model so that the model can achieve better performance under specific task requirements. Adjusting only the target parameters can focus on improving the core accuracy and predictive ability of the model, while adjusting the path selection parameters alone can help optimize the efficiency of data flow and improve the switching effect of the model between different processing networks. When both are adjusted at the same time, the model can strike a balance between accuracy and efficiency to meet diverse task requirements. This flexible adjustment method not only improves the adaptability of the model in various scenarios, but also effectively reduces resource consumption and improves the overall efficiency and stability of the model.

[0111] Further, adjusting the target parameters corresponding to the data processing model and the path selection parameters corresponding to the path selection network in each data encoding layer based on the target sample data until the model training stop condition is reached includes:

[0112] Processing the target sample data based on the data processing model to obtain a first model loss value, and adjusting the target parameter corresponding to the data processing model according to the first model loss value until a first model training stop condition is reached, to obtain a reference data processing model, wherein the first model loss value is a model loss value used to improve the accuracy of a model output result;

[0113] Obtain project sample data for target projects;

[0114] The project sample data is processed based on the data processing model to obtain a second model loss value, and the path selection parameters corresponding to each reference data encoding layer are adjusted according to the second model loss value until the second model training stop condition is reached, wherein the second model loss value is a model loss value used to reduce the gap in selection probability between target data processing networks with similar functions in the model.

[0115] In practical applications, the first model loss value is the loss value used to measure the accuracy of the model output results during training, and the first model training stop condition is the standard for judging whether the training is completed; the reference data processing model is a model adjusted based on the target sample data and the first model loss value; the project sample data is sample data collected for specific tasks or projects; the second model loss value is the loss value for measuring the resource allocation effect of the model when processing project sample data; the second model training stop condition is the standard for judging whether the resource allocation effect has reached stability during training.

[0116] Exemplarily, the first model loss value can be understood as the error calculated by the model when processing the target sample data, which is used to evaluate the accuracy of the output results. During the training process, by continuously adjusting the target parameters to reduce the loss value, the model gradually improves its performance on the data set. For example, in the text generation task, the cross entropy loss function can be used to calculate the difference between the model output and the target text, helping to improve the quality of text generation. Correspondingly, the first model training stop condition can be understood as a criterion for judging whether to stop further adjustment at a specific training stage, which is usually related to the changing trend of the loss value. For example, when the above-mentioned first model loss value no longer decreases significantly after multiple rounds of training, the model reaches the first model training stop condition, indicating that it tends to converge on the current data set. Reasonable stopping conditions help prevent overfitting and save computing resources.

[0117] The reference data processing model can be understood as the model version obtained after the training stop condition is reached. After preliminary adjustment and optimization, it has the ability to process the target data. This model version not only provides high accuracy on the target data, but also lays the foundation for subsequent parameter adjustment and network selection processes to ensure that the data processing system has good performance. Project sample data can be understood as a specific data set used to further evaluate and fine-tune the reference model, which usually contains samples that reflect project requirements. These data simulate the input of the model in actual scenarios, so that it can better adapt to specific needs. For example, in speech recognition tasks, project sample data may include speech data with a specific accent to test the recognition effect of the model under that accent.

[0118] The second model loss value can be understood as the error value used to measure the resource allocation effect when processing project sample data. Unlike the first model loss value, this loss value focuses on the selection accuracy of the path selection network and the expert module to reduce the selection differences between networks with similar functions in the model, thereby improving resource utilization and making multi-task processing more efficient. Correspondingly, the second model training stop condition can be understood as the standard for judging whether the model has reached the convergence state during the resource allocation parameter adjustment process. Usually when the loss value no longer decreases significantly, the model meets the second model training stop condition, thereby ensuring a better balance between task allocation and resource utilization, and improving processing efficiency and stability.

[0119] It should be noted that obtaining the second model loss value by processing the project sample data based on the data processing model can be understood as the difference in selection probabilities between target data processing networks with similar functions in the model calculation model based on the data input. The specific method can be to calculate the difference between the selected path and other potential paths, for example, in the path selection network, compare the selection results of different data processing paths, and then calculate the loss value; or in the expert module, select several more suitable expert modules according to the input data, calculate the weighted loss value between the output generated by these experts and the actual sample data, so as to adjust the path selection parameters; or set the target output based on the task requirements, compare the actual output of the model with the target output, and measure the difference as the loss value, so as to optimize the accuracy and efficiency of resource allocation, etc. This manual does not impose any restrictions on this.

[0120] The target parameters and path selection parameters of the model are adjusted in turn through the first model loss value and the second model loss value, so that the performance of the model on different tasks can be optimized to achieve a balance between accuracy and efficiency. The adjustment of the first model loss value focuses on improving the accuracy of the model, making its predictions on specific data sets more accurate; while the adjustment of the second model loss value focuses on the rationality of resource allocation, and by optimizing the path selection parameters, the model can effectively allocate computing resources in a multi-task environment to avoid excessive use or waste of computing power. This dual optimization strategy can not only ensure the accuracy of the model in task processing, but also improve response speed and computing efficiency, thereby providing better support for the practical application of the model.

[0121] Further, processing the project sample data based on the data processing model to obtain a second model loss value includes:

[0122] Determine a first reference data coding layer, wherein the first reference data coding layer is any one of the reference data coding layers, and the first reference data coding layer includes a path selection network and at least two target data processing networks;

[0123] Inputting the project sample data into the path selection network to obtain path selection information generated by the path selection network;

[0124] Determine at least one first target data processing network and at least one second target data processing network in each target data processing network based on the path selection information, wherein the first target data processing network is a data processing network determined by judging the path selection network to be used for processing the project sample data, and the second target data processing network is a target data processing network other than the first target data processing network in each target data processing network;

[0125] Processing the project sample data according to each first target data processing network to obtain first data characteristic information, and processing the project sample data according to each second target data processing network to obtain second data characteristic information;

[0126] A second model loss value is obtained based on the first data feature information and the second data feature information.

[0127] In actual applications, the path selection information is the processing path information determined by the path selection network according to the input data. The first target data processing network is the main data processing network used to process project sample data, and the first data feature information is the feature information generated by the first target data processing network. The second target data processing network is a backup or auxiliary data processing network, and the second data feature information is the feature information generated by the second target data processing network.

[0128] Path selection information can be understood as the optimal processing path calculated by the path selection network after receiving the input data. This information guides the data flow to the appropriate target data processing network to optimize the processing efficiency. For example, in a multi-expert model, the path selection information may determine that the data flows to the first k expert modules to more efficiently process specific input data and ensure that the task is executed optimally.

[0129] The first target data processing network can be understood as the data processing network determined by the path selection network according to the path selection information. It is the main execution network of the model when processing the input data this time, and is usually selected as the network that better matches the current task requirements. For example, in the natural language processing task, the first target data processing network is the expert module that is more suitable for processing the context. The first data feature information can be understood as the core feature information generated by the first target data processing network, which is the feature information after the data feature information output by each first target data network is fused. This feature information reflects the key information extracted from the input data in the selected network, which is used for further data analysis or decision-making.

[0130] The second target data processing network can be understood as other backup processing networks determined by the path selection network after selecting the first target data processing network. The second data feature information can be understood as feature information after the auxiliary feature information generated by each second target data processing network is fused. This information supplements the main decision of the model and helps improve the comprehensiveness of the processing. For example, in an image recognition task, the second data feature information may contain features of secondary objects to supplement the recognition effect of the main object, thereby improving the overall performance of the model.

[0131] It should be noted that obtaining the second model loss value based on the first data feature information and the second data feature information can be understood as evaluating the resource allocation effect of the model by performing a difference or weighted analysis on the first feature information generated by the network selected this time and the second feature information generated by the network not selected. The specific method can be to calculate the difference between the outputs generated by the first and second data feature information, for example, to compare the similarity of the output features generated by the two, and the difference value can be fed back as the loss value; a weighted method can also be used to weight the first feature information and the second feature information according to the task priority, and the error between the weighted result and the target output is calculated as the loss value; the first and second feature information can also be used to generate different prediction results, and these results can be independently compared with the true value, and the weighted average error can be calculated as a reference for resource allocation optimization, etc. This manual does not impose any restrictions on this.

[0132] By calculating the second model loss value through the first feature information generated by the selected network and the second feature information generated by the unselected network, the load balancing of the model can be effectively achieved, thereby optimizing the efficiency of computing resource utilization. The primary feature information focuses on core processing to ensure that tasks are accurately executed, while the secondary feature information shares part of the work for the primary network when the load is high, making resource allocation more reasonable. By calculating and adjusting the second model loss value, the model can intelligently switch processing loads between different tasks, avoiding performance bottlenecks caused by resource concentration on a few networks, and thus achieving efficient collaboration and load balancing between multiple networks. This mechanism ensures the processing stability and response speed of the model in a multi-task environment.

[0133] The solution of the embodiment of this specification is applied, by introducing multiple reference models and building a variety of target data processing networks based on these reference models, including full data processing networks and compressed data processing networks, which can provide flexible structural configurations during model training. This multi-level data processing network design enables the system to intelligently select a more suitable data processing path according to the needs of specific tasks and training stages, thereby optimizing the efficiency of computing resource utilization. In particular, the combination of the full data processing network and the compressed data processing network not only improves the processing accuracy, but also effectively reduces the model parameters and the amount of calculation, so that the model can achieve a better balance in resource consumption while ensuring high accuracy.

[0134] In addition, by generating a data processing model based on the target data processing network and adopting a path selection network mechanism, the training path can be dynamically adjusted according to the characteristics of different data processing networks. This dynamic path selection not only enhances the adaptability of the model, but also automatically selects a better processing method according to the characteristics of different tasks and training stages, avoiding excessive calculations and redundant resource consumption, thereby effectively improving the speed of training and reasoning. Furthermore, through a multi-level network structure and flexible path selection, it ensures that the model can optimize the use of computing resources under different data samples and task requirements, provide efficient and high-precision solutions, significantly improve computing resource utilization and reduce unnecessary training and reasoning overhead.

[0135] Corresponding to the above method embodiment, this specification also provides a text processing method embodiment, see Figure 3 , Figure 3 A flowchart of a text processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0136] Step 302: Obtain the text to be processed.

[0137] Step 304: input the text to be processed into a data processing model to obtain a text processing result output by the data processing model, wherein the data processing model is trained by the above-mentioned model training method.

[0138] In practical applications, the text to be processed is the original text data that needs to be processed or analyzed in some way, and the text processing result is the output result obtained after processing the text to be processed through the data processing model.

[0139] Exemplarily, the text to be processed can be understood as the input text that needs to be analyzed, converted or optimized in the text processing process. These texts can be content of different sources and types, such as articles, comments, reports, social media posts or any text information that needs further understanding or operation. The text to be processed usually contains raw information, which may include noise, spelling errors or grammatical problems, etc., and needs to be adapted to further analysis or tasks through a specific processing process. For example, in natural language processing tasks, the text to be processed can be a question entered by the user, a paragraph of an article or a document. Processing these texts can help the model understand its meaning and respond.

[0140] The result of text processing can be understood as the output data or information generated after the data processing model processes the text to be processed. This result is usually the key information extracted after model analysis, classification labels, summaries, translated texts, or corrections to the original text. For example, in response to a user's query, the model can convert it into a format that the machine can understand and give a suitable answer, or extract key information from the text for subsequent analysis. In practical applications, the quality and accuracy of text processing results directly affect the effectiveness of subsequent tasks. The results after model processing can be used for a variety of tasks such as information retrieval, sentiment analysis, and text classification.

[0141] The above is a schematic scheme of a text processing method of this embodiment. It should be noted that the technical scheme of the text processing method and the technical scheme of the above model training method belong to the same concept, and the details not described in detail in the technical scheme of the text processing method can be referred to the description of the technical scheme of the above model training method.

[0142] By applying the solution of the embodiments of this specification, by obtaining the text to be processed and inputting it into a carefully trained data processing model, this method can combine the task of preprocessing the text with the designed target data processing network to achieve efficient and accurate text processing results. Since the data processing model can intelligently select the appropriate data processing path through the optimization of the full data processing network and the compressed data processing network during the training process, this method shows higher flexibility and efficiency when processing different types of text.

[0143] Corresponding to the above method embodiment, this specification also provides a text processing method embodiment applied to a cloud device, see Figure 4 , Figure 4 A flowchart of a text processing method applied to a cloud device provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0144] Step 402: receiving a text processing request sent by a terminal-side device, wherein the text processing request includes a text to be processed.

[0145] Step 404: input the text to be processed into a data processing model to obtain a text processing result output by the data processing model, wherein the data processing model is trained by the above-mentioned model training method.

[0146] Step 406: Send the text processing result to the terminal device.

[0147] In actual applications, cloud devices are remote server devices with strong processing capabilities and rich resources, while client devices are devices running locally or on the client side, which usually have smaller processing capabilities. Text processing requests are processing requests initiated from client devices to cloud devices, which usually contain text content that needs to be analyzed or converted.

[0148] Cloud devices can be understood as high-performance server clusters or computing resource pools deployed on cloud computing platforms. These devices usually have powerful processing capabilities, storage capabilities, and network bandwidth, and are suitable for performing complex data processing tasks. For example, cloud devices can be used to run large models, perform data analysis, or provide cloud storage services. In text processing scenarios, cloud devices usually undertake the main computing tasks, receive requests from end-side devices, process large amounts of data, generate processing results, and return the results to the end-side devices. The advantage of cloud devices lies in their resource scalability and efficient processing capabilities, which are suitable for processing complex text analysis and reasoning tasks.

[0149] The end-side device can be understood as a device used locally by the user, which usually has relatively limited computing and storage resources, such as smartphones, laptops, IoT devices, etc. The end-side device is usually used to receive and send data requests and perform relatively simple local processing tasks. The end-side device delegates more complex computing tasks to the cloud for processing by collaborating with cloud devices. For example, a user enters a piece of text on the end-side device, sends a request to the cloud device, and requests the cloud to perform text analysis or translation, and then the cloud returns the analysis results for the user to view. The end-side device plays the role of data collection and request sending in the text processing scenario, reducing the local computing burden.

[0150] A text processing request can be understood as a request initiated by a client device to a cloud device, which usually contains the text data to be processed and the corresponding processing instructions. These requests may be for tasks such as text analysis, natural language understanding, sentiment analysis, and translation. Text processing requests are usually sent in a standardized data format, carrying the text to be processed and the task requirements. The sending of the request is usually triggered automatically by the user or device. After receiving the request, the cloud device will execute the corresponding processing logic according to the requirements and return the processing result. For example, when a user enters a voice text for translation on a smartphone, the phone will send a text processing request to the cloud, and the cloud translation service will process the request and return the translated text.

[0151] The above is a schematic scheme of a text processing method applied to a cloud device in this embodiment. It should be noted that the technical scheme of the text processing method applied to the cloud device belongs to the same concept as the technical scheme of the above text processing method. For details not described in detail in the technical scheme of the text processing method applied to the cloud device, please refer to the description of the technical scheme of the above text processing method.

[0152] By applying the solution of the embodiments of this specification, by receiving the text processing request sent by the terminal device and inputting the text to be processed into the optimized and trained data processing model, this method can provide an efficient and accurate cloud text processing solution. The design of the data processing model is based on a variety of target data processing networks, including full data processing networks and compressed data processing networks. This design enables cloud devices to intelligently select a more suitable data processing path based on different task requirements and actual processing scenarios, thereby effectively reducing the consumption of computing resources while ensuring the accuracy of text processing.

[0153] See also Figure 5 , Figure 5 An architecture diagram of a text processing system provided by an embodiment of the present specification is shown. The text processing system may include a client 100 and a server 200;

[0154] The client 100 is used to send a text processing request including a text to be processed to the server 200;

[0155] The server 200 is used to input the text to be processed into a data processing model, obtain the text processing result output by the data processing model, wherein the data processing model is trained by the above-mentioned model training method; and send the text processing result to the client 100;

[0156] The client 100 is also used to receive the text processing result sent by the server 200.

[0157] By applying the solution of the embodiments of this specification, the text processing system provides an efficient and distributed text processing solution by combining the client and server architectures. The client sends the text to be processed to the server, which processes the text based on a carefully trained data processing model, generates processing results and returns them to the client. This architecture not only improves the scalability of the system, but also optimizes the use of resources, so that text processing tasks can be distributed among different computing nodes, avoiding the bottleneck of a single computing resource.

[0158] The text processing system may include multiple clients 100 and a server 200, wherein the client 100 may be referred to as a terminal device and the server 200 may be referred to as a cloud device. Multiple clients 100 may establish a communication connection through the server 200. In the text processing scenario, the server 200 is used to provide text processing services between multiple clients 100. Multiple clients 100 may serve as a sender or a receiver respectively and realize communication through the server 200.

[0159] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the text processing scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a text processing result according to the data stream and pushes the text processing result to other clients that have established communication.

[0160] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0161] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0162] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0163] It is worth noting that the text processing method provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the text processing method provided in the embodiments of this specification. In other embodiments, the text processing method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0164] The following combination Figure 6 , taking the application of the model training method provided in this specification in the automatic question answering model training as an example, the model training method is further explained. Among them, Figure 6 A processing flow chart of an automatic question-answering model training method provided in an embodiment of the present specification is shown, which specifically includes the following steps.

[0165] Step 602: Obtain at least two reference models, and obtain at least two target data processing networks based on the reference models, where the target data processing networks include a full data processing network and a compressed data processing network.

[0166] In practical applications, by obtaining the above-mentioned multiple reference models as expert models, the foundation can be laid for the subsequent construction of the target automatic question answering model.

[0167] Step 604: Determine a reference model data processing layer in the reference model, and obtain a target data processing network according to the reference model data processing layer.

[0168] Specifically, in this process, by obtaining the structure of the data processing network in the reference model, it can be ensured that the obtained network is a more important network in the expert model (ie, the reference model), avoiding excessive structural redundancy.

[0169] Step 606: Obtain initial parameter information corresponding to each model data processing layer, and selectively generate a full data processing network and a compressed data processing network based on the initial parameter information.

[0170] Step 608: Select a full data processing network and at least two compressed data processing networks according to the pre-configuration to construct the data encoding layers of the automatic question answering model.

[0171] Step 610: Obtain target sample data, and adjust the parameters of each module in the automatic question-answering model based on the target sample data.

[0172] Step 612: Obtain sample data related to the target project, and obtain the model load balancing loss value corresponding to each data encoding layer of the above-mentioned trained automatic question-answering model based on the target sample data.

[0173] In practical applications, the model load balancing loss value is used to reduce the gap in selection probabilities between target data processing networks with similar functions in the model.

[0174] Step 614: Train the parameters of the path selection network in each data encoding layer of the automatic question answering model using the load balancing loss value described above until the model training stop condition is reached, so that the trained automatic question answering model improves the accuracy of the answers generated for the target project.

[0175] By applying the solution of the embodiments of this specification, by introducing multiple reference models and building a diverse target data processing network based on these models, this method provides a flexible and efficient network architecture for the training of automatic question-answering models. In this process, the combination of the full data processing network and the compressed data processing network not only enhances the processing accuracy of the model, but also effectively reduces the consumption of computing resources. In particular, extracting important network structures in the reference model and combining the initial parameter information of the data processing layer can avoid unnecessary redundant structures and improve the structural simplicity and computational efficiency of the model.

[0176] In addition, the intelligent selection and generation of the target data processing network enables the automatic question-answering model to flexibly adjust the network structure according to different data samples and task requirements. Through the path selection network mechanism, the model can dynamically adjust the data processing path and optimize the use of computing resources during training. This flexible path selection not only avoids excessive calculation and resource waste, but also significantly improves the speed of training and reasoning, ensuring high accuracy while greatly reducing redundant calculations.

[0177] Through this multi-level, dynamically adjusted data processing network structure, this method effectively improves the adaptability of the automatic question answering model. In particular, during the training process for the target project, the model load balancing loss value is used to optimize the network selection, ensuring that the model can answer specific questions more accurately, and optimizing the use of computing resources, avoiding performance bottlenecks and over-computing, and achieving an efficient and high-precision automatic question answering solution.

[0178] Corresponding to the above method embodiment, this specification also provides a model training device embodiment, Figure 7 FIG. 1 shows a schematic diagram of a model training device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0179] An acquisition module 702 is configured to acquire at least two reference models, and acquire at least two target data processing networks based on the reference models, wherein the target data processing networks include at least one full data processing network and at least one compressed data processing network;

[0180] A generating module 704 is configured to generate a data processing model according to each target data processing network, wherein the data processing model includes at least one data encoding layer, and the data encoding layer includes at least one full data processing network and at least one compressed data processing network;

[0181] The training module 706 is configured to obtain target sample data and train the data processing model based on the target sample data until the data processing model reaches a model training stop condition, wherein the target sample data is data used to train the data processing model.

[0182] Optionally, the acquisition module 702 is further configured to:

[0183] Determine a target reference model, wherein the target reference model is any one of the reference models;

[0184] A reference model data processing layer is determined in the target reference model, and a target data processing network is acquired according to the reference model data processing layer.

[0185] Optionally, the acquisition module 702 is further configured to:

[0186] Acquire initial parameter information corresponding to the reference model data processing layer;

[0187] According to the initial parameter information, obtaining parameter characteristic information corresponding to the reference model data processing layer;

[0188] A compressed data processing network is generated based on the parameter characteristic information.

[0189] Optionally, the acquisition module 702 is further configured to:

[0190] Based on the initial parameter information, obtaining parameter scaling characteristic information and parameter output characteristic information;

[0191] Parameter characteristic information is acquired according to the parameter scaling characteristic information and the parameter output characteristic information.

[0192] Optionally, the acquisition module 702 is further configured to:

[0193] Acquire initial parameter information corresponding to the reference model data processing layer;

[0194] The initial parameter information is determined to be a full data processing network.

[0195] Optionally, the generating module 704 is further configured to:

[0196] Determining a target full data processing network and at least one target compressed data processing network in each target data processing network, wherein the target full data processing network is any one of the full data processing networks, and the target compressed data processing network is any one of the compressed data processing networks;

[0197] Acquire a target data coding layer based on the target full data processing network and each target compressed data processing network, wherein the target data coding layer is any one of the data coding layers;

[0198] A data processing model is generated according to each target data encoding layer.

[0199] Optionally, the generating module 704 is further configured to:

[0200] Acquire a path selection network, wherein the path selection network is used to select at least one data processing network for data processing in this round from among the target full data processing network and each target compressed data processing network;

[0201] A target data coding layer is obtained according to the path selection network, the target full data processing network and each target compressed data processing network.

[0202] Optionally, the training module 706 is further configured to:

[0203] Based on the target sample data, the target parameters corresponding to the data processing model and / or the path selection parameters corresponding to the path selection network in each data encoding layer are adjusted until the data processing model reaches the model training stop condition.

[0204] Optionally, the training module 706 is further configured to:

[0205] Processing the target sample data based on the data processing model to obtain a first model loss value, and adjusting the target parameter corresponding to the data processing model according to the first model loss value until a first model training stop condition is reached, to obtain a reference data processing model, wherein the first model loss value is a model loss value used to improve the accuracy of a model output result;

[0206] Obtain project sample data for target projects;

[0207] The project sample data is processed based on the data processing model to obtain a second model loss value, and the path selection parameters corresponding to each reference data encoding layer are adjusted according to the second model loss value until the second model training stop condition is reached, wherein the second model loss value is a model loss value used to reduce the gap in selection probability between target data processing networks with similar functions in the model.

[0208] Optionally, the training module 706 is further configured to:

[0209] Determine a first reference data coding layer, wherein the first reference data coding layer is any one of the reference data coding layers, and the first reference data coding layer includes a path selection network and at least two target data processing networks;

[0210] Inputting the project sample data into the path selection network to obtain path selection information generated by the path selection network;

[0211] Determine at least one first target data processing network and at least one second target data processing network in each target data processing network based on the path selection information, wherein the first target data processing network is a data processing network determined by the path selection network to be used for processing the project sample data, and the second target data processing network is a target data processing network other than the first target data processing network in each target data processing network;

[0212] Processing the project sample data according to each first target data processing network to obtain first data characteristic information, and processing the project sample data according to each second target data processing network to obtain second data characteristic information;

[0213] A second model loss value is obtained based on the first data feature information and the second data feature information.

[0214] By applying the solution of the embodiments of this specification, this method realizes the function of flexibly configuring the data processing structure during the model training process by introducing multiple reference models and generating diversified target data processing networks (including full data processing networks and compressed data processing networks) based on these reference models. This multi-level structural design enables the system to dynamically select a more suitable processing path according to the needs of different tasks and training stages, thereby optimizing the use of computing resources and improving training efficiency. In particular, the combination of the full data processing network and the compressed data processing network not only improves the processing accuracy, but also significantly reduces the number of parameters and computational complexity of the model, so that the model can greatly reduce resource consumption while ensuring high accuracy.

[0215] The above is a schematic scheme of a model training device of this embodiment. It should be noted that the technical scheme of the model training device and the technical scheme of the above-mentioned model training method belong to the same concept, and the details not described in detail in the technical scheme of the model training device can be referred to the description of the technical scheme of the above-mentioned model training method.

[0216] Corresponding to the above method embodiment, this specification also provides a cloud training platform embodiment. Figure 8 FIG. 1 shows a schematic diagram of the structure of a cloud training platform provided by an embodiment of the present specification. Figure 8 As shown, the cloud training platform includes a request interface and a response unit;

[0217] The request interface 802 is used to receive a task generation request sent by a terminal device;

[0218] The response unit 804 obtains a data processing model based on the task generation request, wherein the data processing model is trained by the above-mentioned model training method; and generates task information based on the data processing model, wherein the task information is used for the terminal device to perform the text processing task.

[0219] Optionally, the response unit 804 is further configured to:

[0220] The task generation request is parsed to obtain request information, and a data processing model is obtained based on the request information.

[0221] By applying the solution of the embodiments of this specification, by introducing a flexible request interface and response unit in the cloud training platform, the method can quickly respond and provide a data processing model required for a suitable text processing task after receiving a task generation request from a terminal device. This design not only improves the response speed of task processing, but also intelligently selects and optimizes data processing models when receiving different requests. Specifically, by combining a diverse target data processing network, the platform can select a more suitable processing path based on the characteristics of the request information to ensure the efficiency and accuracy of text processing.

[0222] The above is a schematic scheme of a cloud training platform of this embodiment. It should be noted that the technical scheme of the cloud training platform belongs to the same concept as the technical scheme of the above-mentioned model training method and antibody generation method. The details not described in detail in the technical scheme of the cloud training platform can be referred to the description of the technical scheme of the above-mentioned model training method.

[0223] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0224] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0225] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0226] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0227] Among them, the processor 920 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned model training method, text processing method, and text processing method applied to cloud devices.

[0228] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the model training method, text processing method and text processing method applied to cloud devices belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the model training method, text processing method and text processing method applied to cloud devices.

[0229] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned model training method, text processing method, and text processing method applied to a cloud device.

[0230] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the model training method, text processing method and text processing method applied to cloud devices belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the model training method, text processing method and text processing method applied to cloud devices.

[0231] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned model training method, text processing method, and text processing method applied to a cloud device.

[0232] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the model training method, text processing method and text processing method applied to cloud devices belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the model training method, text processing method and text processing method applied to cloud devices.

[0233] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0234] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0235] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0236] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0237] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A model training method, comprising: Acquire at least two reference models, and acquire at least two target data processing networks based on the reference models, wherein the target data processing networks include at least one full data processing network and at least one compressed data processing network; Determine a target full data processing network and at least one target compressed data processing network in each target data processing network; obtain a path selection network, and obtain a target data coding layer according to the path selection network, the target full data processing network and each target compressed data processing network; generate a data processing model according to at least one target data coding layer, wherein the target full data processing network is any one of the full data processing networks, the target compressed data processing network is any one of the compressed data processing networks, the path selection network is used to select at least one data processing network for data processing in this round among the target full data processing network and each target compressed data processing network, and the data processing model is used for text processing; Obtain target sample data and project sample data for a target project, and obtain a first model loss value based on the target sample data; determine a first reference data coding layer, wherein the first reference data coding layer is any one of the reference data coding layers, and the first reference data coding layer includes a path selection network and at least two target data processing networks; input the project sample data into the path selection network, and obtain path selection information generated by the path selection network; determine at least one first target data processing network and at least one second target data processing network in each target data processing network based on the path selection information, wherein the first target data processing network is determined by the path selection network to be used for processing the project a data processing network for the target sample data, wherein the second target data processing network is a target data processing network other than the first target data processing network in each target data processing network; processing the project sample data according to each first target data processing network to obtain first data feature information, and processing the project sample data according to each second target data processing network to obtain second data feature information; obtaining a second model loss value based on the first data feature information and the second data feature information; adjusting the target parameters corresponding to the data processing model and the path selection parameters corresponding to the path selection network in each data encoding layer according to the first model loss value and the second model loss value until the data processing model reaches the model training stop condition, wherein the target sample data is the data used to train the data processing model.

2. The method of claim 1, wherein obtaining at least two target data processing networks based on each reference model comprises: Determine a target reference model, wherein the target reference model is any one of the reference models; A reference model data processing layer is determined in the target reference model, and a target data processing network is acquired according to the reference model data processing layer.

3. The method according to claim 2, wherein obtaining a target data processing network according to the reference model data processing layer comprises: Acquire initial parameter information corresponding to the reference model data processing layer; According to the initial parameter information, obtaining parameter characteristic information corresponding to the reference model data processing layer; A compressed data processing network is generated based on the parameter characteristic information.

4. The method according to claim 3, obtaining parameter characteristic information corresponding to the reference model data processing layer according to the initial parameter information, comprising: Based on the initial parameter information, obtaining parameter scaling characteristic information and parameter output characteristic information; Parameter characteristic information is acquired according to the parameter scaling characteristic information and the parameter output characteristic information.

5. The method according to claim 2, wherein obtaining a target data processing network according to the reference model data processing layer comprises: Acquire initial parameter information corresponding to the reference model data processing layer; The initial parameter information is determined to be a full data processing network.

6. The method of claim 1, adjusting the target parameters corresponding to the data processing model and the path selection parameters corresponding to the path selection network in each data encoding layer according to the first model loss value and the second model loss value until the data processing model reaches the model training stop condition, comprising: According to the first model loss value, the target parameter corresponding to the data processing model is adjusted until the first model training stop condition is reached, so as to obtain a reference data processing model, wherein the first model loss value is a model loss value used to improve the accuracy of the model output result; The path selection parameters corresponding to each reference data encoding layer are adjusted according to the second model loss value until the second model training stop condition is reached, wherein the second model loss value is a model loss value used to reduce the difference in selection probability between target data processing networks with similar functions in the model.

7. A text processing method, comprising: Get the text to be processed; The text to be processed is input into a data processing model to obtain a text processing result output by the data processing model, wherein the data processing model is trained by the method described in any one of claims 1 to 6.

8. A text processing method, applied to a cloud device, comprising: A text processing request sent by a receiving terminal device, wherein the text processing request includes a text to be processed; Inputting the text to be processed into a data processing model to obtain a text processing result output by the data processing model, wherein the data processing model is trained by the method according to any one of claims 1 to 6; The text processing result is sent to the terminal device.

9. A training platform, comprising a request interface and a response unit; The request interface is used to receive a task generation request sent by a terminal device; The response unit, based on the task generation request, obtains a data processing model, wherein the data processing model is trained by the method described in any one of claims 1-6; based on the data processing model, generates task information, wherein the task information is used for the terminal device to perform a text processing task.

10. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images