Model processing method and device, electronic equipment, storage medium and computer program product

Abstract

The invention discloses a model processing method and device, electronic equipment, a storage medium and a computer program product. The method comprises the steps that first feature data is acquired, and the first feature data comprises at least two types of sample data associated with a first task; using the first feature data to determine first information corresponding to at least two regions associated with a first model, the first information representing an importance degree of each region in the at least two regions to each category of sample data in the at least two categories of sample data, the at least two areas are at least used for knowledge learning; using the first information to determine a growth strategy of the first model, the growth strategy being used for indicating a to-be-grown area in the at least two areas; based on a knowledge retention algorithm, a second model is obtained through the growth strategy and the first model, and the second model is used for executing the first task.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model processing method, device, electronic device, storage medium and computer program product. Background Art

[0002] With the continuous breakthroughs in artificial intelligence and natural language processing technologies, large models based on the Transformer architecture (such as large language models (LLM)) have shown good performance in different task scenarios. Since LLM needs to acquire knowledge from massive amounts of data, training LLM often requires a lot of time and computing resources. With the continuous emergence of new data, periodic retraining of LLM based on new data will result in a waste of resources and time. At the same time, since the distribution of new data is usually different from that of old data, learning new knowledge from new data may affect the old knowledge acquired previously, resulting in a sharp decline in the performance of LLM for tasks corresponding to the old knowledge.

[0003] In response to the above problems, the relevant technology proposes a solution that uses model growth to enable LLM to continue learning. In this way, LLM can inherit the old knowledge learned previously on the basis of continuous learning, thereby avoiding catastrophic forgetting of old knowledge. At the same time, it further improves the ability of LLM to quickly adapt to new knowledge, effectively saving time and loss of computing resources.

[0004] However, in related model growth schemes, there is a problem that the performance of the grown model on the task is not good. Summary of the invention

[0005] To solve related technical problems, the embodiments of the present application provide a model processing method, device, electronic device, storage medium and computer program product.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application embodiment provides a model processing method, including:

[0008] Acquire first feature data, wherein the first feature data includes sample data of at least two categories associated with the first task;

[0009] Determine, by using the first feature data, first information corresponding to at least two regions associated with the first model, wherein the first information represents the importance of each of the at least two regions to each category of sample data of the at least two categories, wherein the at least two regions are at least used for knowledge learning;

[0010] Determine a growth strategy of the first model using the first information, wherein the growth strategy is used to indicate a region to be grown in the at least two regions;

[0011] Based on a knowledge retention algorithm, a second model is obtained by utilizing the growth strategy and the first model, and the second model is used to perform the first task.

[0012] In the above solution, the step of using the first feature data to determine first information corresponding to at least two regions associated with the first model includes:

[0013] Using the first feature data, respectively determine second information and third information corresponding to the at least two regions, the second information representing the degree of deviation of each region of the at least two regions affected by the sample data of each category of the at least two categories of sample data, and the third information representing the degree of contribution of each region of the at least two regions to the output result of the first model;

[0014] The first information is obtained by using the second information and the third information.

[0015] In the above solution, the using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes:

[0016] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0017] The second information is determined using the second data included in the first feature data, the first model and the trained first model.

[0018] In the above solution, the determining the second information by using the second data included in the first feature data, the first model and the trained first model includes:

[0019] Inputting the second data into the first model to obtain a first output result corresponding to the at least two regions;

[0020] Inputting the second data into the trained first model to obtain a second output result corresponding to the at least two regions;

[0021] The second information is obtained by using the first output result and the second output result.

[0022] In the above solution, the using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes:

[0023] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0024] The third information is determined using the second data included in the first feature data and the trained first model.

[0025] In the above solution, the determining the third information by using the second data included in the first feature data and the trained first model includes:

[0026] In the process of adjusting the parameters corresponding to the at least two regions, the second data is input into the trained first model to obtain eighth information, wherein the eighth information includes gradient information of the output result of the first model relative to each of the at least two regions;

[0027] The third information is obtained by using the eighth information.

[0028] In the above solution, the using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes:

[0029] In a case where the at least two regions include a first region and a second region, respectively determining a plurality of first computing units associated with the first region and a plurality of second computing units associated with the second region;

[0030] Determine, by using the first feature data, fourth information corresponding to the plurality of first computing units and fifth information corresponding to the plurality of second computing units, and obtain the second information by using the fourth information and the fifth information, wherein the fourth information represents the degree of deviation of each first computing unit in the plurality of first computing units affected by the sample data of each category in the at least two categories of sample data, and the fifth information represents the degree of deviation of each second computing unit in the plurality of second computing units affected by the sample data of each category in the at least two categories of sample data;

[0031] Using the first characteristic data, determine the sixth information corresponding to the multiple first computing units and the seventh information corresponding to the multiple second computing units, and use the sixth information and the seventh information to obtain the third information, the sixth information representing the contribution degree of each first computing unit among the multiple first computing units to the execution result of the first task, and the seventh information representing the contribution degree of each second computing unit among the multiple second computing units to the execution result of the first task.

[0032] In the above solution, the second model is obtained by using the growth strategy and the first model based on the knowledge retention algorithm, including:

[0033] When the region to be grown in the at least two regions is the first region, based on the knowledge retention algorithm, the number of attention heads corresponding to the first region is increased to obtain the second model;

[0034] When the region to be grown among the at least two regions is the second region, the dimension corresponding to the second region is increased based on a knowledge retention algorithm to obtain the second model.

[0035] In the above scheme, the method further comprises:

[0036] Based on the knowledge retention algorithm, the number of layers of the first model is increased to obtain an updated second model.

[0037] In the above scheme, the method further comprises:

[0038] The second model is trained using the first feature data.

[0039] The present application also provides a model processing device, including:

[0040] An acquisition unit, configured to acquire first feature data, wherein the first feature data includes sample data of at least two categories associated with the first task;

[0041] a first determining unit, configured to determine, by using the first feature data, first information corresponding to at least two regions associated with a first model, wherein the first information represents the importance of each of the at least two regions to each category of sample data of the at least two categories, wherein the at least two regions are at least used for knowledge learning;

[0042] A second determining unit, configured to determine a growth strategy of the first model by using the first information, wherein the growth strategy is used to indicate a region to be grown in the at least two regions;

[0043] A growing unit is used to obtain a second model based on a knowledge retention algorithm using the growth strategy and the first model, where the second model is used to perform the first task.

[0044] The embodiment of the present application also provides an electronic device, comprising: a processor and a communication interface; wherein:

[0045] The communication interface is used to obtain first feature data, wherein the first feature data includes sample data of at least two categories associated with the first task;

[0046] The processor is used to determine first information corresponding to at least two areas associated with a first model using the first feature data, the first information characterizing the importance of each of the at least two areas to each category of sample data in the at least two categories, and the at least two areas are at least used for knowledge learning; determine a growth strategy of the first model using the first information, the growth strategy being used to indicate areas to be grown in the at least two areas; and obtain a second model using the growth strategy and the first model based on a knowledge retention algorithm, the second model being used to perform the first task.

[0047] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0048] Wherein, the processor is used to execute the steps of any of the above-mentioned model processing methods when running the computer program.

[0049] An embodiment of the present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model processing methods are implemented.

[0050] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned model processing methods when executed by a processor.

[0051] The model processing method, device, electronic device, storage medium and computer program product provided by the embodiment of the present application obtain first feature data, the first feature data includes at least two categories of sample data associated with the first task; using the first feature data, determining first information corresponding to at least two regions associated with the first model, the first information characterizing the importance of each region in the at least two regions to each category of sample data in the at least two categories of sample data, the at least two regions are at least used for knowledge learning; using the first information, determining the growth strategy of the first model, the growth strategy is used to indicate the region to be grown in the at least two regions; based on the knowledge retention algorithm, using the growth strategy and the first model, obtaining a second model, the second model is used to perform the first task. The technical solution provided by the embodiment of the present application can obtain the influence and expression of knowledge in different regions of the model by quantifying the importance of the region of the model to various types of sample data, and then, using the quantified results, more reasonably selecting the expansion region corresponding to the model, and realizing the growth of the model based on the principle of knowledge retention, so that on the basis of maximizing the retention of the original knowledge of the model, the knowledge contained in the new data can be more effectively expressed in the model, thereby improving the overall performance of the grown model for the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a structural schematic diagram of LLM regional distribution in the related art;

[0053] Figure 2 A schematic diagram of a method flow chart for model processing in an embodiment of the present application;

[0054] Figure 3 This is a schematic diagram of the structure of the first computing unit and the second computing unit in the embodiment of the present application;

[0055] Figure 4 This is a schematic diagram of the structure of the region growth in the embodiment of the present application;

[0056] Figure 5 A schematic diagram of the process of model generation for the example growth system used in this application;

[0057] Figure 6 A schematic flow chart of a method for generating quantitative results for an example growth system for the application of this application;

[0058] Figure 7 This is a schematic diagram of the structure of the model processing device in the embodiment of the present application;

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0061] In the relevant model growth scheme, the old parameters of the large model (such as LLM, specifically GPT-4, PaLM 2 and LLaMA2) are initialized, and the capacity of one or more areas of the model is gradually expanded (which can also be understood as increasing), such as the dimension of the feed-forward neural network (FFN), the number of multi-head attention (MHA), the number of layers (which can be expressed as Layer number in English), etc., so that the previously learned knowledge can be inherited to avoid catastrophic forgetting.

[0062] However, the above model growth scheme does not associate the expanded area with the data features. Specifically, in the model growth scenario, as new data continues to emerge, it may affect the feature distribution of the data, that is, different categories of data have different feature distributions. For example, code data usually has strict grammatical rules and structures, which makes the distribution characteristics of this type of data relatively fixed and predictable; while natural language data is more flexible and contains rich semantic information. There are many combinations of words and phrases, which makes the feature distribution of this type of data more complex and diverse. Figure 1 As shown in Figure 1, different areas of the human brain are responsible for different knowledge, and the model will imitate the distribution of neurons in the human brain. Therefore, specific areas of the model may be more sensitive to specific data. However, in related model growth strategies, it is often assumed that pre-training follows the same data distribution, and there is no selection and design of expansion areas for different types of data.

[0063] Regarding the storage mechanism and interpretability of the model's internal knowledge, related studies have confirmed that similar knowledge expressions in the model rely on the same core position, that is, inhibiting or activating the corresponding position will significantly affect the expression of the corresponding knowledge. However, since the model uses complex nonlinear interactions in densely connected layers, research on the interpretability of large models only focuses on the neuron and parameter dimensions, and it is impossible to understand the internal mechanism of the model's regional storage of knowledge, and thus it is impossible to use the internal mechanism to guide the model's growth area to improve model performance.

[0064] In addition, although model growth is an important research direction, related growth strategies do not achieve the goal of function retention or knowledge retention very well. That is, the model may change the output of old data, thereby losing some previously learned knowledge or patterns. The loss of original knowledge may cause the performance of the model on old data to deteriorate, and it may even be necessary to retrain the model after growth to restore performance.

[0065] In summary, the relevant model growth schemes do not take into account the influence and expression of knowledge in different areas of the model, resulting in the model growth strategy not paying attention to the connection between the expansion area and different feature data. As a result, the grown model cannot adapt to changes in data distribution, thus affecting the performance of the task.

[0066] Based on this, in various embodiments of the present application, by quantifying the importance of the knowledge area of ​​the model (such as MHA or FFN area), the focus is on the relationship between the internal regional dimensions of the model and the feature data of different categories; the quantification results are used to guide the model to expand areas for different categories of data, and to achieve growth while retaining model knowledge before and after expansion, so that the model can efficiently and continuously learn new knowledge while retaining the original knowledge, thereby improving the overall performance of the model and the effects of downstream tasks.

[0067] The present application embodiment provides a model processing method, such as Figure 2 As shown, applied to an electronic device, the method includes:

[0068] Step 201: Acquire first feature data, where the first feature data includes sample data of at least two categories associated with a first task;

[0069] Step 202: using the first feature data, determining first information corresponding to at least two regions associated with the first model, wherein the first information represents the importance of each of the at least two regions to each category of sample data of the at least two categories, wherein the at least two regions are at least used for knowledge learning;

[0070] Step 203: using the first information, determining a growth strategy of the first model, wherein the growth strategy is used to indicate a region to be grown in the at least two regions;

[0071] Step 204: Based on the knowledge retention algorithm, using the growth strategy and the first model, a second model is obtained, and the second model is used to perform the first task.

[0072] In actual application, the electronic device may include a device capable of executing steps 201 to 204, such as a server, a laptop, a tablet computer, or a desktop computer. In addition, the first model may be called a large language model, a large model, or an original base model, etc. The first model may include a model based on the Transformer architecture, such as an LLM. The embodiment of the present application does not limit the type of the first model as long as its function is realized.

[0073] In actual application, in step 201, the electronic device can obtain the first feature data from the network or from a database. The first feature data can be understood as sample data (also called feature data) associated with the first task (such as video data detection service or image classification service, etc., the embodiment of the present application does not limit the type of the first task). The first feature data may include at least two categories of pre-labeled sample data.

[0074] Then, the electronic device can use a quantization algorithm to respectively calculate the first information corresponding to each of the at least two areas to quantify the influence and expression of knowledge in different areas; wherein, the at least two areas can be understood as areas associated with knowledge learning, such as MHA areas or FFN areas, and the first information can be understood as the importance score of the area for the expression of sample data of different categories.

[0075] Specifically, in one embodiment, the specific implementation of step 202 may include:

[0076] Using the first feature data, respectively determine second information and third information corresponding to the at least two regions, the second information representing the degree of deviation of each region of the at least two regions affected by the sample data of each category of the at least two categories of sample data, and the third information representing the degree of contribution of each region of the at least two regions to the output result of the first model;

[0077] The first information is obtained by using the second information and the third information.

[0078] Among them, the second information may include the offset score (also called distance score) of each region affected by the sample data of each category; the third information can be understood as the contribution degree of the region to knowledge prediction, that is, the contribution score to the correct output result of the first model (also called attribution score).

[0079] In actual application, for the second information, the electronic device can use the first feature data to pre-train the first model, and determine the second information based on the pre-trained first model and the first model.

[0080] Specifically, in one embodiment, the using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes:

[0081] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0082] The second information is determined using the second data included in the first feature data, the first model and the trained first model.

[0083] In actual application, the electronic device can pre-divide the first feature data to obtain the divided first data and the second data; wherein the first data can be called training data, at least used to train the first model; the second data can be called verification data, at least as input data for the first model and the trained first model, so as to obtain the second information and the third information.

[0084] In actual application, during the training process of the first model, the electronic device may input the first data into the first model to obtain the trained first model; wherein the first data may be represented as Data, and the first model (i.e., the original model before training) may be represented as M origin , the first model after training can be expressed as M data .

[0085] Then, the electronic device can input the second data into the first model and the trained first model respectively, and calculate the second information corresponding to each area based on the quantization algorithm to evaluate the impact of the second data on different areas.

[0086] Specifically, in one embodiment, determining the second information by using the second data included in the first feature data, the first model, and the trained first model includes:

[0087] Inputting the second data into the first model to obtain a first output result corresponding to the at least two regions;

[0088] Inputting the second data into the trained first model to obtain a second output result corresponding to the at least two regions;

[0089] The second information is obtained by using the first output result and the second output result.

[0090] The first output result may include a third output result of sample data of each region for each category, and the second output result may include a fourth output result of sample data of each region for each category.

[0091] In actual application, before inputting the second data, the electronic device may divide each area into finer granularity, that is, define the minimum computing unit of each area, and respectively determine the importance of the minimum computing unit in each area for different categories of sample data.

[0092] Specifically, in one embodiment, the using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes:

[0093] In a case where the at least two regions include a first region and a second region, respectively determining a plurality of first computing units associated with the first region and a plurality of second computing units associated with the second region;

[0094] Determine, by using the first feature data, fourth information corresponding to the plurality of first computing units and fifth information corresponding to the plurality of second computing units, and obtain the second information by using the fourth information and the fifth information, wherein the fourth information represents the degree of deviation of each first computing unit in the plurality of first computing units affected by the sample data of each category in the at least two categories of sample data, and the fifth information represents the degree of deviation of each second computing unit in the plurality of second computing units affected by the sample data of each category in the at least two categories of sample data;

[0095] Using the first characteristic data, determine the sixth information corresponding to the multiple first computing units and the seventh information corresponding to the multiple second computing units, and use the sixth information and the seventh information to obtain the third information, the sixth information representing the contribution degree of each first computing unit among the multiple first computing units to the execution result of the first task, and the seventh information representing the contribution degree of each second computing unit among the multiple second computing units to the execution result of the first task.

[0096] Among them, the first area may include an MHA area, and accordingly, the first computing unit can be understood as a pair in the attention score matrix (which can be expressed as Attention score in English) of each attention head in the multiple attention heads corresponding to the first area, and the first computing unit can characterize the correlation between the current position and other positions in the matrix; the second area may include an FFN area, and accordingly, the second computing unit can be understood as a neuron corresponding to the second area.

[0097] In actual application, in the process of determining the multiple first calculation units, the electronic device may determine multiple attention heads corresponding to the first area; for each attention head, an attention score is determined to obtain the multiple first calculation units, and one first calculation unit is an element in the attention score; wherein the attention score A can be determined by the following formula:

[0098]

[0099] Wherein, Q represents the query vector (expressed in English as ) input to the first region, which can represent the words or features that need to be paid attention to at the current position; K T Represented as a key vector, it can represent words or features at other positions; d k Represented as the dimension of the key vector; A can be a square matrix of s*s, where s represents the input length of the model, and the first computing unit a ij Represented as the element in row i and column j in the attention score.

[0100] In addition, for the multiple second computing units, the electronic device can determine multiple linear transformation layers and activation functions corresponding to the second area, and then determine the fifth output result after being processed by the first linear transformation layer and the activation function, and a second computing unit is an element in the fifth output result.

[0101] For example, Figure 3 As shown, in the process of training the first model, the electronic device can use the first data as input data to obtain the sixth output result corresponding to the MHA area, thereby determining the multiple first calculation units; wherein the first data can include a query vector and a key vector, and the sixth output result includes the attention score of each attention head in the first area. In addition, the electronic device can input the first data into the FFN area to obtain the fifth output result corresponding to the FFN area; wherein the FFN area usually includes two main linear transformation layers (represented as W l1 &b b1 and W l2 &b b2 ) and a nonlinear activation function (which can be expressed as GELU), the fifth output result is the output result after being processed by the first linear transformation layer and the activation function, and can be used as the input of the second linear transformation layer. The fifth output result N can be expressed as:

[0102] N=GELU(H×W l1 +b l1 ) (2)

[0103] Among them, H can be represented as the input data of the FFN area, and the second computing unit n i is any neuron in N.

[0104] In actual application, for each first calculation unit associated with the first area, the electronic device can input the sample data of each category into the first area in the first model, and at the same time, suppress the first calculation units in turn (which can also be understood as setting the first calculation units to 0), so as to obtain the seventh output result of the first area associated with the first calculation unit respectively; wherein the third output result of the first area for each category of sample data can include the seventh output result of the first area associated with each first calculation unit.

[0105] In addition, the electronic device can also input the sample data of each category into the first area in the trained first model, and at the same time, suppress the first calculation units in turn, so as to obtain the eighth output results of the first area associated with the first calculation units respectively; wherein the fourth output result of the first area for each category of sample data may include the eighth output result of the first area associated with each first calculation unit.

[0106] Then, using the seventh output result and the eighth output result, the fourth information corresponding to each first calculation unit associated with the first area can be determined, and the fourth information may include the offset score of each first calculation unit affected by the sample data of each category; wherein, the fourth information can be obtained by calculating the distance between the seventh output result and the eighth calculation result.

[0107] For example, for the first computing unit a ij The electronic device can input the sample data of category A as input data to the first model and the trained first model respectively, and at the same time, suppress the first calculation unit a ij , to obtain the seventh output result and the eighth output result; by cosine similarity, the fourth information is calculated; wherein the fourth information Dis_score MHA (a ij ) can be obtained by the following formula:

[0108]

[0109] in, It is expressed as: for input data X, when the first computing unit a ij When set to 0, the seventh output result of the first area; It is expressed as: for input data X, when the first computing unit a ij When set to 0, the eighth output result of the first area.

[0110] In actual application, for each second computing unit associated with the second area, the electronic device can input the sample data of each category into the second area in the first model, and at the same time, suppress the second computing units in turn, so as to obtain the ninth output result of the second area associated with the second computing unit respectively; wherein the third output result of the second area for each category of sample data can include the ninth output result of the second area associated with each second computing unit.

[0111] In addition, the electronic device can also input the sample data of each category into the second area in the trained first model, and at the same time, suppress the second computing units in turn, so as to obtain the tenth output result of the second area associated with the second computing unit respectively; wherein the fourth output result of the second area for each category of sample data may include the tenth output result of the second area associated with each second computing unit.

[0112] Then, using the ninth output result and the tenth output result, the fifth information corresponding to each second computing unit can be determined, and the fifth information may include the offset score of each second computing unit affected by the sample data of each category; wherein, it can be obtained by calculating the distance between the ninth output result and the tenth output result.

[0113] For example, for the second computing unit a ij The electronic device can input the sample data of category B as input data to the first model and the trained first model respectively, and at the same time, suppress the second calculation unit a ij , to obtain the ninth output result and the tenth output result; by cosine similarity, the fifth information is calculated; wherein the fifth information Dis_score FFN (a ij ) can be obtained by the following formula:

[0114]

[0115] in, It is expressed as: for input data X, when the second computing unit a ij When set to 0, the ninth output result of the second area; It is expressed as: for input data X, when the second computing unit a ij When set to 0, the tenth output result of the second area.

[0116] It should be noted that, when inputting sample data of the same category, if the degree of offset represented by the fourth information is greater, it means that the characteristic changes of the first area are obvious, that is, the first computing unit has a greater impact on the first model when performing the first task; if the degree of offset represented by the fifth information is greater, it means that the characteristic changes of the second area are obvious, that is, the second computing unit has a greater impact on the first model when performing the first task.

[0117] In actual application, for the third information, the electronic device can use the first data to pre-train the first model, so as to determine the third information based on the pre-trained first model.

[0118] Specifically, in one embodiment, the using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes:

[0119] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0120] The third information is determined using the second data included in the first feature data and the trained first model.

[0121] Here, the electronic device may input sample data of each category into different regions in turn based on the knowledge attribution method of integrated gradient to evaluate the contribution of each region to task execution.

[0122] Specifically, in one embodiment, the determining the third information by using the second data included in the first feature data and the trained first model includes:

[0123] In the process of adjusting the parameters corresponding to the at least two regions, the second data is input into the trained first model to obtain eighth information, wherein the eighth information includes gradient information of the output result of the first model relative to each of the at least two regions;

[0124] The third information is obtained by using the eighth information.

[0125] In actual application, for the first area, the electronic device may input sample data of each category in the second data into the trained first model to obtain the probability that the trained first model can output the correct execution result of the first task, wherein the probability of the correct execution result can be expressed as:

[0126]

[0127] Among them, Y * Indicates the correct execution result. Represents a given constant value assigned to the first computation unit.

[0128] Next, in order to determine the sixth information corresponding to each first computing unit, the electronic device may adjust the parameters corresponding to each first computing unit in turn. In this process, the electronic device may respectively integrate the gradient of the output result of the trained first model relative to each first computing unit, and then obtain the sixth information. The sixth information may include the contribution score of each first computing unit to the execution result of the first task (specifically, the correct execution result); wherein the sixth information Attr_score MHA (a ij ) can be expressed as:

[0129]

[0130] in, Represented as the output result of the first model after training relative to the first computing unit a ij Gradient of Attr_score MHA (a ij ) is expressed as the process in which the parameter s of the first computing unit is adjusted from 0 to 1, by a ij The degree of change in the probability of outputting the correct execution result caused by the change, if the first computing unit a ij The expression of the execution result of the first task has a great influence, the gradient will be significant, and then Attr_score MHA (a ij ) is larger.

[0131] In practical application, since it is difficult to continuously integrate the output result of the trained first model relative to the gradient of the first computing unit, the electronic device may determine the sixth information based on formula (6) using the Riemann approximation method; wherein the sixth information may be obtained using the following formula:

[0132]

[0133] Here, m represents the number of approximation steps, for example, it can be 20.

[0134] In actual application, after obtaining the sixth information corresponding to each first computing unit, the electronic device may combine the fourth information and the sixth information corresponding to the plurality of first computing units to obtain the ninth information, wherein the ninth information may include the importance score of each first computing unit for each category of sample data, wherein the importance score is associated with the importance degree, and the importance score may be obtained based on formula (3) and formula (7):

[0135] Final_score MHA (a ij )=Norm(Dis_score MHA (a ij )*Attr_score MHA (a ij )) (8)

[0136] That is, for each first computing unit, by multiplying the offset score and contribution score associated with the first computing unit and normalizing the result of the multiplication, the electronic device can obtain the importance score of the first computing unit for each category of sample data.

[0137] Here, when the importance scores corresponding to multiple first calculation units are determined, the electronic device can set a first threshold (the value can be set as needed, and the embodiment of the present application is not limited to this), and compare the importance score corresponding to each first calculation unit with the first threshold; if the importance score corresponding to the first calculation unit is greater than or equal to the first threshold, the electronic device can record the first calculation unit as an important first calculation unit, and then, using the number of important first calculation units and the number of first calculation units, obtain tenth information, wherein the tenth information includes a first ratio of the number of important first calculation units to the number of first calculation units; wherein, the larger the value of the first ratio, the greater the degree to which the first area is affected by the sample data of each category, that is, it is more sensitive to knowledge.

[0138] In actual application, for the second area, the electronic device can input sample data of each category in the second data into the trained first model, and refer to formula (5) to obtain the probability that the trained first model can output the correct execution result of the first task.

[0139] Next, in order to determine the seventh information corresponding to each second computing unit, the electronic device may adjust the parameters corresponding to each second computing unit in turn. In this process, the electronic device may respectively integrate the gradient of the output result of the trained first model relative to each second computing unit, and then obtain the seventh information. The seventh information may include the contribution score of each second computing unit to the execution result of the first task (specifically, the correct execution result); wherein the seventh information Attr_score FFA (a ij ) can be expressed as:

[0140]

[0141] In actual application, after obtaining the seventh information corresponding to each first computing unit, the electronic device may combine the fifth information and the seventh information corresponding to the plurality of second computing units to obtain the eleventh information, wherein the eleventh information may include the importance score of each second computing unit for each category of sample data, wherein the importance score is associated with the importance degree, and the importance score Final_score FFN (a ij ) can be obtained based on formula (4) and formula (9):

[0142] Final_score FFN (a ij )=Norm(Dis_score FFN (a ij )*Attr_score FFN (a ij ))(10)

[0143] Here, when the importance scores corresponding to multiple second calculation units are determined, the electronic device can set a second threshold (the value can be set as needed, and the embodiment of the present application is not limited to this), and compare the importance score corresponding to each second calculation unit with the second threshold; if the importance score corresponding to the second calculation unit is greater than or equal to the second threshold, the electronic device can record the second calculation unit as an important second calculation unit, and then, using the number of important second calculation units and the number of second calculation units, obtain twelfth information, and the twelfth information includes a second ratio of the number of important second calculation units to the number of second calculation units; wherein, the larger the value of the second ratio, the greater the degree to which the second area is affected by the sample data of each category, that is, it is more sensitive to knowledge.

[0144] In actual application, in step 203, the electronic device can determine the growth strategy using the tenth information and the twelfth information. Specifically, for a category of sample data, if the first ratio is greater than the second ratio, the electronic device can determine that the corresponding area to be grown is the first area; if the first ratio is less than the second ratio, the electronic device can determine that the area to be grown is the second area.

[0145] For example, Figure 4 As shown, for the sample data of category C, if the first ratio is greater than the second ratio, then the electronic device can determine the subsequent algorithm-based knowledge retention to initialize the expanded model MHA area to achieve the growth of the first model; otherwise, the electronic device can determine the subsequent algorithm-based knowledge retention to initialize the expanded model FFN area to achieve the growth of the first model.

[0146] In an embodiment of the present application, the importance score of the region is used to guide the growth strategy of the first model, so that the performance of the grown first model after training can be further improved, that is, new knowledge is learned while retaining old knowledge, so that the learned knowledge can be maximized.

[0147] In actual application, in order to enable the first model to retain old learned knowledge, the electronic device can achieve the growth of the first model on the basis of knowledge retention.

[0148] Specifically, in one embodiment, the specific implementation of step 204 may include:

[0149] When the region to be grown in the at least two regions is the first region, based on the knowledge retention algorithm, the number of attention heads corresponding to the first region is increased to obtain the second model;

[0150] When the region to be grown among the at least two regions is the second region, the dimension corresponding to the second region is increased based on a knowledge retention algorithm to obtain the second model.

[0151] In actual application, when the area to be grown is the first area, the electronic device may add one or more attention heads based on the knowledge retention algorithm to achieve the growth of the first area, thereby obtaining the second model; wherein the weight matrix corresponding to each attention head may include a key vector (which can be expressed as W K ), query vector (which can be expressed as W Q ) and the value vector (which can be represented as W V ), the hyperparameter b corresponding to the attention head is used to control the dimensional scaling of the number of attention heads.

[0152] Here, when the number of attention heads increases from b1 to b2, the electronic device may keep the hyperparameters of the original attention heads unchanged (which may also be understood as fixed), and at the same time, assign corresponding hyperparameters to the new attention heads; wherein, it may be expressed as:

[0153]

[0154] in, It is represented as the b-th attention head, and S is represented as a random number.

[0155] It should be noted that the principle of the knowledge retention algorithm is: by performing parameter replication during initialization, the relevant modules of the first model are initially scaled to a larger size, and the second model can maintain its performance in processing previous data. At the same time, the second model can be directly trained with new data later without going through a period of low performance. For example, for the first model (which can be represented as M origin (X,θ origin ), X represents the input data of the first model, θ origin denoted as the parameters of the first model), by denoting the parameters of the second model (which can be denoted as M enlarge (X,θ enlarge )) Select a new set of parameters so that the second model can retain the knowledge of the first model, and the selected parameters can satisfy the following formula:

[0156]

[0157] In actual application, after the growth of the first region is realized, the electronic device can initialize the relevant parameters (such as the weight matrix) of the first region after the growth to ensure that for the same input data, the execution result output by the second model is the same as the execution result output by the first model. O ) generates the final result (which can be expressed as H MHA ), therefore, the electronic device can set all the parts corresponding to the newly added attention heads in the linear matrix to 0 to achieve initialization of function retention.

[0158] In actual application, when the area to be grown is the second area, the electronic device can increase the hidden dimension (also called internal dimension) of the second area based on the knowledge retention algorithm to achieve the growth of the second area, thereby obtaining the second model; wherein the hyperparameter f corresponding to the second area is used to control the scaling of the hidden dimension.

[0159] Exemplarily, when the number of hidden dimensions increases from f1 to f2, the electronic device may l2 The corresponding expansion weights are set to an all-zero matrix, and W l1 and b b1 The expansion weights are randomly initialized to ensure that the output of the second model after growth is the same as that of the first model for the same input data, so as to achieve function-preserving initialization.

[0160] In actual application, by increasing the number of attention heads or the dimension corresponding to the second area, the width inside the model can be increased; in order to balance the size of the model, the electronic device can also grow the number of layers based on the knowledge retention algorithm, thereby increasing the length inside the model.

[0161] Based on this, in one embodiment, the method may further include:

[0162] Based on the knowledge retention algorithm, the number of layers of the first model is increased to obtain an updated second model.

[0163] Here, for the first model, the electronic device can select one or more network layers from the first model, copy the selected network layers, and insert the copied network layers as new network layers into the first model, thereby increasing the number of layers.

[0164] Exemplarily, for the L-layer model, the electronic device can randomly select network layer l for copying to obtain a newly added network layer, and then insert the copied network layer l' between the original network layer l and the subsequent network layer l+1, so that the second model can be updated.

[0165] In practical applications, for the increased number of layers, the electronic device can also use the idea of ​​residual links to skip the copy layer by setting the weight λ of the newly added network layer to 1, thereby achieving initialization with function preservation; wherein the output result of the newly added network layer can be expressed as:

[0166] H l '=lH l +(1-l)H l ' (13)

[0167] Among them, H l Represented as the output of the original network layer, H l ' represents the output result of the newly added network layer; in the subsequent training process, the weight λ will gradually change from 1 to 0, and the residual connection will disappear, thereby enabling the increase of the number of model layers.

[0168] In actual application, after the growth of the model is realized, the electronic device can use the first feature data to pre-train the updated second model to achieve continuous learning of the model.

[0169] Based on this, in one embodiment, the method may further include:

[0170] The second model is trained using the first feature data.

[0171] It should be noted that, since the areas to be grown may be different for different categories of sample data, the electronic device can grow and train the model for one category of sample data, and then grow and train the model for another category of sample data, so that continuous learning of the model can be achieved.

[0172] The model processing method provided in the embodiment of the present application obtains first feature data, the first feature data includes at least two categories of sample data associated with the first task; using the first feature data, determining first information corresponding to at least two regions associated with the first model, the first information characterizing the importance of each region in the at least two regions to each category of sample data in the at least two categories, and the at least two regions are at least used for knowledge learning; using the first information, determining the growth strategy of the first model, the growth strategy is used to indicate the region to be grown in the at least two regions; based on the knowledge retention algorithm, using the growth strategy and the first model, obtaining a second model, the second model is used to perform the first task. The technical solution provided in the embodiment of the present application can obtain the influence and expression of knowledge in different regions of the model by quantifying the importance of the region of the model to various types of sample data, and then, using the quantified results, more reasonably select the expansion region corresponding to the model, and realize the growth of the model based on the principle of knowledge retention, so that on the basis of maximizing the retention of the original knowledge of the model, it can ensure that the knowledge contained in the new data is more effectively expressed in the model, thereby improving the overall performance of the trained model for the task.

[0173] The present application is further described in detail below in conjunction with embodiments.

[0174] In the application example of this application, a growth system of a large language model based on knowledge area is proposed; Figure 5 As shown in the figure, in the first stage, a module for quantifying the importance of knowledge regions within the model is designed to calculate the importance scores of new data expressed by MHA and FFN regions for continuous training of various features, so as to evaluate the impact of new data on different regions of the model and provide guidance for the growth strategy of knowledge retention of large models in the second stage; in the second stage, according to the guidance of regional importance scores obtained in the first stage, two different expansion methods, MHA head expansion or FFN dimension expansion, are adopted, and a knowledge-retaining model growth algorithm is used to realize the growth of the model, that is, on the premise of maximizing the retention of the original knowledge, different growth regions are dynamically designed for different data.

[0175] Specifically, in the first stage, the growth system generates quantitative results of the importance of different regions through the quantification module, such as Figure 6 As shown, the following steps are included:

[0176] Step 601: Determine the minimum calculation unit of the MHA area (i.e., the first calculation unit mentioned above);

[0177] Among them, the minimum calculation unit is a pair of attention score matrices of each attention head.

[0178] Step 602: Calculate the distance score (ie, the fourth information mentioned above);

[0179] Among them, the distance score represents the degree of deviation of the MHA area under the influence of this type of data.

[0180] Here, the quantization module uses the training data (i.e., the first data) in the feature data (i.e., the first feature data) to train the original base model (i.e., the first model) based on the Transformer architecture, and obtains the trained model (i.e., the first model after training). Then, the verification data (i.e., the second data) is input into the original model and the trained model respectively; in the process of inputting the verification data into the original model and the trained model respectively, the quantization module calculates the output result of the MHA region when the minimum computational unit is suppressed; wherein, for the same verification data as input X, the output vector of the MHA region is used as output. Then, the distance between the two vectors is calculated, and the calculated distance is determined as the distance score corresponding to the minimum computational unit.

[0181] Step 603: Calculate the attribution score (ie, the sixth information mentioned above);

[0182] Here, the quantization module calculates the attribution score based on the knowledge attribution method of integrated gradient; wherein, the verification data is used as input data, and the probability change of the correct output result of the MHA area caused by the change of the minimum computing unit is calculated by gradually changing the parameter value corresponding to the minimum computing unit to obtain the attribution score corresponding to the minimum computing unit.

[0183] Step 604: Calculate the final importance score;

[0184] Among them, the final importance score is the combination of the distance score and the attribution score.

[0185] In practical applications, the quantification module calculates the product of the distance score and the attribution score corresponding to the minimum calculation unit of the MHA area to obtain the final importance score.

[0186] Step 605: Determine whether the final importance score is greater than a threshold;

[0187] If the final importance score is greater than or equal to the threshold, execute step 606; otherwise, execute step 607.

[0188] Step 606: Determine important units;

[0189] Here, the quantization module may record the minimum calculation unit as an important unit, and determine to update the number of important units.

[0190] Step 607: Whether to traverse all minimum computing units;

[0191] If all minimum calculation units are traversed, step 608 is executed; otherwise, step 601 is executed, that is, the final importance score of the next minimum calculation unit is calculated.

[0192] Step 608: Calculate the importance ratio (ie, the first information mentioned above);

[0193] Here, the quantification module calculates the importance ratio of the MHA area based on the number of important units and the number of all minimum computational units.

[0194] Step 609: Determine the minimum calculation unit of the FFN area (i.e., the second calculation unit mentioned above);

[0195] Among them, the smallest computing unit is the neuron in the FFN area.

[0196] Step 610: Calculate the distance score (ie, the fifth information mentioned above);

[0197] Here, in the process of inputting the verification data into the original model and the trained model respectively, the quantization module calculates the output result of the FFN area when the minimum computational unit is suppressed; wherein, for the same verification data as input X, the output vector of the FFN area is used as the output. Then, the distance between the two vectors is calculated, and the calculated distance is determined as the distance score of the minimum computational unit.

[0198] Step 611: Calculate the attribution score (i.e. the seventh information mentioned above);

[0199] Here, the quantization module calculates the attribution score based on the knowledge attribution method of integrated gradient; wherein, the verification data is used as input data, and the probability change of the correct output result of the FFN area caused by the change of the minimum computing unit is calculated by gradually changing the parameter value corresponding to the minimum computing unit to obtain the attribution score corresponding to the minimum computing unit.

[0200] Step 612: Calculate the final importance score;

[0201] In actual application, the quantization module calculates the product of the distance score and the attribution score corresponding to the minimum calculation unit of the FFN area to obtain the final importance score.

[0202] Step 613: Determine whether the final importance score is greater than a threshold;

[0203] If the final importance score is greater than or equal to the threshold, execute step 614; otherwise, execute step 615.

[0204] Step 614: Determine important units;

[0205] Here, the quantization module may record the minimum calculation unit as an important unit, and determine to update the number of important units.

[0206] Step 615: Whether to traverse all minimum computing units;

[0207] If all minimum calculation units are traversed, step 616 is executed; otherwise, step 609 is executed, that is, the final importance score of the next minimum calculation unit is calculated.

[0208] Step 616: Calculate the importance ratio (ie, the first information mentioned above).

[0209] Here, the quantization module calculates the importance ratio of the FFN area based on the number of important units and the number of all minimum computing units.

[0210] Next, the growth system determines the specific growth strategy of the model, that is, the growth area of ​​the model, according to the quantified results of the importance of different areas and based on the principle of knowledge retention; if for a type of data, the importance ratio of the MHA area is higher than that of the FFN area, the MHA area of ​​the model is initialized and expanded based on the knowledge retention algorithm, and the number of layers is expanded to achieve the growth of the width and depth of the model; if the importance ratio of the MHA area is lower than that of the FFN area, the FFN area is expanded, and the number of layers is expanded to achieve the growth of the width and depth of the model.

[0211] Then, the growth system uses the data of this type to pre-train the grown model (ie, the second model mentioned above) to achieve continuous learning of the model.

[0212] In the embodiment of the present application, by designing a quantification module of the importance of knowledge regions and quantifying the importance scores of different regions to various types of feature data, it is possible to evaluate the impact of new data on different regions of the model, thereby dynamically adjusting the model growth strategy to adapt to changes in data distribution.

[0213] Secondly, by using the knowledge-retaining initialization algorithm, the model can retain the original knowledge while growing. In this way, as new data is continuously introduced, the model can continue to absorb and learn new knowledge through pre-training on new data. In other words, the model has the ability to continuously learn, thereby improving the overall performance of the model and the effects of downstream tasks. At the same time, the trained model can not only retain the knowledge that has been learned, but also adapt to new task requirements, making the model more adaptable.

[0214] In addition, the growth system proposed in the application example of this application is highly flexible and scalable, can be adjusted and optimized according to actual application needs, is suitable for models of different scales and types and different data, and has good universality and practicality.

[0215] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a model processing device, which is arranged on an electronic device, such as Figure 7 As shown, the device comprises:

[0216] An acquisition unit 701 is configured to acquire first feature data, where the first feature data includes sample data of at least two categories associated with a first task;

[0217] A first determining unit 702 is configured to determine, using the first feature data, first information corresponding to at least two regions associated with the first model, wherein the first information represents the importance of each of the at least two regions to each category of sample data of the at least two categories, and the at least two regions are at least used for knowledge learning;

[0218] A second determining unit 703 is used to determine a growth strategy of the first model by using the first information, where the growth strategy is used to indicate a region to be grown in the at least two regions;

[0219] The growing unit 704 is used to obtain a second model based on a knowledge retention algorithm using the growing strategy and the first model, where the second model is used to perform the first task.

[0220] In one embodiment, the first determining unit 702 is configured to:

[0221] Using the first feature data, respectively determine second information and third information corresponding to the at least two regions, the second information representing the degree of deviation of each region of the at least two regions affected by the sample data of each category of the at least two categories of sample data, and the third information representing the degree of contribution of each region of the at least two regions to the output result of the first model;

[0222] The first information is obtained by using the second information and the third information.

[0223] In one embodiment, the first determining unit 702 is configured to:

[0224] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0225] The second information is determined using the second data included in the first feature data, the first model and the trained first model.

[0226] In one embodiment, the first determining unit 702 is configured to:

[0227] Inputting the second data into the first model to obtain a first output result corresponding to the at least two regions;

[0228] Inputting the second data into the trained first model to obtain a second output result corresponding to the at least two regions;

[0229] The second information is obtained by using the first output result and the second output result.

[0230] In one embodiment, the first determining unit 702 is configured to:

[0231] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0232] The third information is determined using the second data included in the first feature data and the trained first model.

[0233] In one embodiment, the first determining unit 702 is configured to:

[0234] In the process of adjusting the parameters corresponding to the at least two regions, the second data is input into the trained first model to obtain eighth information, wherein the eighth information includes gradient information of the output result of the first model relative to each of the at least two regions;

[0235] The third information is obtained by using the eighth information.

[0236] In one embodiment, the first determining unit 702 is configured to:

[0237] In a case where the at least two regions include a first region and a second region, respectively determining a plurality of first computing units associated with the first region and a plurality of second computing units associated with the second region;

[0238] Determine, by using the first feature data, fourth information corresponding to the plurality of first computing units and fifth information corresponding to the plurality of second computing units, and obtain the second information by using the fourth information and the fifth information, wherein the fourth information represents the degree of deviation of each first computing unit in the plurality of first computing units affected by the sample data of each category in the at least two categories of sample data, and the fifth information represents the degree of deviation of each second computing unit in the plurality of second computing units affected by the sample data of each category in the at least two categories of sample data;

[0239] Using the first characteristic data, determine the sixth information corresponding to the multiple first computing units and the seventh information corresponding to the multiple second computing units, and use the sixth information and the seventh information to obtain the third information, the sixth information representing the contribution degree of each first computing unit among the multiple first computing units to the execution result of the first task, and the seventh information representing the contribution degree of each second computing unit among the multiple second computing units to the execution result of the first task.

[0240] In one embodiment, the growth unit 704 is used to:

[0241] When the region to be grown in the at least two regions is the first region, based on the knowledge retention algorithm, the number of attention heads corresponding to the first region is increased to obtain the second model;

[0242] When the region to be grown among the at least two regions is the second region, the dimension corresponding to the second region is increased based on a knowledge retention algorithm to obtain the second model.

[0243] In one embodiment, the growth unit 704 is further used for:

[0244] Based on the knowledge retention algorithm, the number of layers of the first model is increased to obtain an updated second model.

[0245] In one embodiment, the growth unit 704 is further used for:

[0246] The second model is trained using the first feature data.

[0247] In actual application, the acquisition unit 701 can be implemented by a communication interface in the model processing device; the first determination unit 702, the second determination unit 703 and the growth unit 704 can be implemented by a processor in the model processing device.

[0248] It should be noted that: the model processing device provided in the above embodiment only uses the division of the above program modules as an example when performing model processing. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the model processing device provided in the above embodiment and the model processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0249] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 8 As shown, the electronic device 800 includes:

[0250] Communication interface 801, capable of exchanging information with other devices;

[0251] A processor 802, connected to the communication interface 801 to implement information interaction with other devices, and used to execute the method provided by one or more of the above technical solutions when running a computer program;

[0252] A memory 803 , on which the computer program is stored.

[0253] Specifically, the communication interface 801 is used to obtain first feature data, where the first feature data includes sample data of at least two categories associated with the first task;

[0254] The processor 802 is used to determine first information corresponding to at least two areas associated with the first model using the first feature data, the first information characterizing the importance of each of the at least two areas to each category of sample data in the at least two categories, and the at least two areas are at least used for knowledge learning; determine a growth strategy of the first model using the first information, the growth strategy being used to indicate areas to be grown in the at least two areas; and obtain a second model using the growth strategy and the first model based on a knowledge retention algorithm, the second model being used to perform the first task.

[0255] In one embodiment, the processor 802 is configured to:

[0256] Using the first feature data, respectively determine second information and third information corresponding to the at least two regions, the second information representing the degree of deviation of each region of the at least two regions affected by the sample data of each category of the at least two categories of sample data, and the third information representing the degree of contribution of each region of the at least two regions to the output result of the first model;

[0257] The first information is obtained by using the second information and the third information.

[0258] In one embodiment, the processor 802 is configured to:

[0259] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0260] The second information is determined using the second data included in the first feature data, the first model and the trained first model.

[0261] In one embodiment, the processor 802 is configured to:

[0262] Inputting the second data into the first model to obtain a first output result corresponding to the at least two regions;

[0263] Inputting the second data into the trained first model to obtain a second output result corresponding to the at least two regions;

[0264] The second information is obtained by using the first output result and the second output result.

[0265] In one embodiment, the processor 802 is configured to:

[0266] Using the first data included in the first feature data, training the first model to obtain a trained first model;

[0267] The third information is determined using the second data included in the first feature data and the trained first model.

[0268] In one embodiment, the processor 802 is configured to:

[0269] In the process of adjusting the parameters corresponding to the at least two regions, the second data is input into the trained first model to obtain eighth information, wherein the eighth information includes gradient information of the output result of the first model relative to each of the at least two regions;

[0270] The third information is obtained by using the eighth information.

[0271] In one embodiment, the processor 802 is configured to:

[0272] In a case where the at least two regions include a first region and a second region, respectively determining a plurality of first computing units associated with the first region and a plurality of second computing units associated with the second region;

[0273] Determine, by using the first feature data, fourth information corresponding to the plurality of first computing units and fifth information corresponding to the plurality of second computing units, and obtain the second information by using the fourth information and the fifth information, wherein the fourth information represents the degree of deviation of each first computing unit in the plurality of first computing units affected by the sample data of each category in the at least two categories of sample data, and the fifth information represents the degree of deviation of each second computing unit in the plurality of second computing units affected by the sample data of each category in the at least two categories of sample data;

[0274] Using the first characteristic data, determine the sixth information corresponding to the multiple first computing units and the seventh information corresponding to the multiple second computing units, and use the sixth information and the seventh information to obtain the third information, the sixth information representing the contribution degree of each first computing unit among the multiple first computing units to the execution result of the first task, and the seventh information representing the contribution degree of each second computing unit among the multiple second computing units to the execution result of the first task.

[0275] In one embodiment, the processor 802 is configured to:

[0276] When the region to be grown in the at least two regions is the first region, based on the knowledge retention algorithm, the number of attention heads corresponding to the first region is increased to obtain the second model;

[0277] When the region to be grown among the at least two regions is the second region, the dimension corresponding to the second region is increased based on a knowledge retention algorithm to obtain the second model.

[0278] In one embodiment, the processor 802 is further configured to:

[0279] Based on the knowledge retention algorithm, the number of layers of the first model is increased to obtain an updated second model.

[0280] In one embodiment, the processor 802 is further configured to:

[0281] The second model is trained using the first feature data.

[0282] It should be noted that the specific processing process of the processor 802 and the communication interface 801 can be understood by referring to the above method.

[0283] Of course, in actual application, the various components in the electronic device 800 are coupled together through the bus system 804. It can be understood that the bus system 804 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as bus system 804 .

[0284] The memory 803 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 800. Examples of such data include: any computer program used to operate on the electronic device 800.

[0285] The method disclosed in the above embodiment of the present application can be applied to the processor 802, or implemented by the processor 802. The processor 802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 802 or an instruction in the form of software. The above-mentioned processor 802 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 802 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 803, and the processor 802 reads the information in the memory 803 and completes the steps of the above method in combination with its hardware.

[0286] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0287] It can be understood that the memory (memory 803) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0288] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 803 storing a computer program, and the computer program can be executed by a processor 802 of an electronic device 800 to complete the aforementioned model processing method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0289] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 802 of the electronic device 800 to complete the steps of the aforementioned model processing method.

[0290] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0291] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0292] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A model processing method, characterized in that: include: Acquire first feature data, wherein the first feature data includes sample data of at least two categories associated with the first task; Determine, by using the first feature data, first information corresponding to at least two regions associated with the first model, wherein the first information represents the importance of each of the at least two regions to each category of sample data of the at least two categories, wherein the at least two regions are at least used for knowledge learning; Determine a growth strategy of the first model using the first information, wherein the growth strategy is used to indicate a region to be grown in the at least two regions; Based on a knowledge retention algorithm, a second model is obtained by utilizing the growth strategy and the first model, and the second model is used to perform the first task.

2. The method according to claim 1, characterized in that The using the first feature data to determine first information corresponding to at least two regions associated with the first model includes: Using the first feature data, respectively determine second information and third information corresponding to the at least two regions, the second information representing the degree of deviation of each region of the at least two regions affected by the sample data of each category of the at least two categories of sample data, and the third information representing the degree of contribution of each region of the at least two regions to the output result of the first model; The first information is obtained by using the second information and the third information.

3. The method according to claim 2, characterized in that The using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes: Using the first data included in the first feature data, training the first model to obtain a trained first model; The second information is determined using the second data included in the first feature data, the first model and the trained first model.

4. The method according to claim 3, characterized in that The determining the second information by using the second data included in the first feature data, the first model and the trained first model includes: Inputting the second data into the first model to obtain a first output result corresponding to the at least two regions; Inputting the second data into the trained first model to obtain a second output result corresponding to the at least two regions; The second information is obtained by using the first output result and the second output result.

5. The method according to claim 2, characterized in that: The using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes: Using the first data included in the first feature data, training the first model to obtain a trained first model; The third information is determined using the second data included in the first feature data and the trained first model.

6. The method according to claim 5, characterized in that The determining the third information by using the second data included in the first feature data and the trained first model includes: In the process of adjusting the parameters corresponding to the at least two regions, the second data is input into the trained first model to obtain eighth information, wherein the eighth information includes gradient information of the output result of the first model relative to each of the at least two regions; The third information is obtained by using the eighth information.

7. The method according to claim 2, characterized in that The using the first feature data to respectively determine the second information and the third information corresponding to the at least two regions includes: In a case where the at least two regions include a first region and a second region, respectively determining a plurality of first computing units associated with the first region and a plurality of second computing units associated with the second region; Determine, by using the first feature data, fourth information corresponding to the plurality of first computing units and fifth information corresponding to the plurality of second computing units, and obtain the second information by using the fourth information and the fifth information, wherein the fourth information represents the degree of deviation of each first computing unit in the plurality of first computing units affected by the sample data of each category in the at least two categories of sample data, and the fifth information represents the degree of deviation of each second computing unit in the plurality of second computing units affected by the sample data of each category in the at least two categories of sample data; Using the first characteristic data, determine the sixth information corresponding to the multiple first computing units and the seventh information corresponding to the multiple second computing units, and use the sixth information and the seventh information to obtain the third information, the sixth information representing the contribution degree of each first computing unit among the multiple first computing units to the execution result of the first task, and the seventh information representing the contribution degree of each second computing unit among the multiple second computing units to the execution result of the first task.

8. The method according to claim 1, characterized in that: The method of obtaining a second model based on the knowledge retention algorithm by using the growth strategy and the first model includes: When the region to be grown in the at least two regions is the first region, based on the knowledge retention algorithm, the number of attention heads corresponding to the first region is increased to obtain the second model; When the region to be grown among the at least two regions is the second region, the dimension corresponding to the second region is increased based on a knowledge retention algorithm to obtain the second model.

9. The method according to claim 8, characterized in that The method further comprises: Based on the knowledge retention algorithm, the number of layers of the first model is increased to obtain an updated second model.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: The second model is trained using the first feature data.

11. A model processing device, characterized in that: include: An acquisition unit, configured to acquire first feature data, wherein the first feature data includes sample data of at least two categories associated with the first task; a first determining unit, configured to determine, by using the first feature data, first information corresponding to at least two regions associated with a first model, wherein the first information represents the importance of each of the at least two regions to each category of sample data of the at least two categories, wherein the at least two regions are at least used for knowledge learning; A second determining unit, configured to determine a growth strategy of the first model by using the first information, wherein the growth strategy is used to indicate a region to be grown in the at least two regions; A growing unit is used to obtain a second model based on a knowledge retention algorithm using the growth strategy and the first model, where the second model is used to perform the first task.

12. An electronic device, characterized in that: include: Processor and communication interface; wherein, The communication interface is used to obtain first feature data, wherein the first feature data includes sample data of at least two categories associated with the first task; The processor is used to determine first information corresponding to at least two areas associated with a first model using the first feature data, the first information characterizing the importance of each of the at least two areas to each category of sample data in the at least two categories, and the at least two areas are at least used for knowledge learning; determine a growth strategy of the first model using the first information, the growth strategy being used to indicate areas to be grown in the at least two areas; and obtain a second model using the growth strategy and the first model based on a knowledge retention algorithm, the second model being used to perform the first task.

13. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 10.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images