Model data processing method and electronic equipment

By determining and applying the hierarchy and transformation functions of the target neural network model on the user equipment side, the model parameters are transformed, and the security and adaptability problems of model parameter deployment are solved, and reasonable and secure deployment are realized on the user equipment side.

CN118940230APending Publication Date: 2024-11-12LENOVO (BEIJING) LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410993148.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When deploying model parameters of neural network models on the user equipment side, there is a risk of leakage, and the demand for model parameters varies greatly in different application scenarios. How to reasonably deploy model parameters becomes a challenge.

Method used

By determining the target neural network model to be protected and its hierarchy, applying model protection requests to determine the target layer and its transformation functions that need to be protected, transforming the model parameters, reducing the risk of leakage, and based on the computing power and business categories of the user equipment Reasonable deployment of information.

Benefits of technology

It effectively reduces the risk of model parameters leakage, improves the security of model parameters, and reasonably deploys model parameters according to the specific needs of user equipment to meet the needs of different application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940230A_ABST
    Figure CN118940230A_ABST
Patent Text Reader

Abstract

The invention discloses a model data processing method and electronic equipment, and the method comprises the steps: determining a to-be-protected target neural network model based on a model protection request, and the target neural network model comprises a plurality of layers; determining at least one target layer to be protected in the plurality of layers and transformation functions corresponding to the model parameters in the target layer; based on a transformation function corresponding to the model parameter in the target layer, performing parameter transformation on the model parameter to obtain a transformed model parameter; and deploying the transformed model parameters in each target layer to a user equipment side.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model data processing method and electronic device. Background Art

[0002] With the continuous development of artificial intelligence technology, more and more neural network models are being deployed and run on user devices.

[0003] In different application scenarios, the user device's requirements for neural network models vary greatly; moreover, when deploying the model parameters of the neural network model on the user device, the risk of model parameter leakage also needs to be considered. Therefore, how to reasonably deploy the model parameters on the user device is a problem that technical personnel in this field need to solve. Summary of the invention

[0004] On the one hand, the present application provides a model data processing method, comprising:

[0005] Based on the model protection request, determining a target neural network model to be protected, wherein the target neural network model includes multiple layers;

[0006] Determine at least one target layer to be protected among the multiple layers and a transformation function corresponding to each model parameter in the target layer;

[0007] Based on the transformation function corresponding to the model parameters in the target layer, the model parameters are transformed to obtain transformed model parameters;

[0008] The transformed model parameters in each of the target layers are deployed to the user device end.

[0009] In a possible implementation manner, determining at least one target layer to be protected among the multiple layers and a transformation function corresponding to each model parameter in the target layer includes:

[0010] Based on the model category of the target neural network model, at least one target layer to be protected among the multiple layers and a transformation function corresponding to each model parameter in the target layer are determined.

[0011] In another possible implementation, the determining, based on the model category of the target neural network model, at least one target layer to be protected among the multiple layers and a transformation function corresponding to each model parameter in the target layer includes:

[0012] In response to the target neural network model being a first type neural network model, at least an attention layer in the target neural network model is determined as a target layer to be protected, and a first transformation function corresponding to each first parameter matrix in the attention layer and a second transformation function corresponding to each first normalization parameter in the attention layer are determined, wherein the first transformation function includes a first permutation matrix and a set interference coefficient, and the second transformation function includes the first permutation matrix;

[0013] In response to the target neural network model being a second type of neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and a third transformation function corresponding to each second parameter matrix in the attention layer and a fourth transformation function corresponding to each second normalization parameter in the attention layer are determined, wherein the third transformation function and the fourth transformation function include a second permutation matrix.

[0014] In another possible implementation, if the target neural network model is a first type neural network model, the first parameter matrix in the attention layer includes a first sub-parameter matrix, a second sub-parameter matrix, and a third sub-parameter matrix;

[0015] The performing parameter transformation on the model parameters based on the transformation function corresponding to the model parameters in the target layer includes:

[0016] For a first sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the first sub-parameter matrix, the first sub-parameter matrix is ​​transformed using a random number and a transposed matrix of a first permutation matrix;

[0017] For a second sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the second sub-parameter matrix, transform the second sub-parameter matrix using a random number, a transposed matrix of the first permutation matrix, and a first reversible matrix;

[0018] For the third sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the third sub-parameter matrix, the third sub-parameter matrix is ​​transformed using random numbers, a first permutation matrix and an inverse matrix corresponding to the first reversible matrix.

[0019] In another possible implementation, if the target neural network model is a first type neural network model, the first parameter matrix in the attention layer includes a first sub-parameter matrix, a second sub-parameter matrix, and a third sub-parameter matrix;

[0020] The performing parameter transformation on the model parameters based on the transformation function corresponding to the model parameters in the target layer includes:

[0021] For a first sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the first sub-parameter matrix, the first sub-parameter matrix is ​​transformed using a transposed matrix of the first permutation matrix and a third permutation matrix, and the transformed first sub-parameter matrix is ​​secondarily transformed in combination with a confusion matrix;

[0022] For a second sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the second sub-parameter matrix, transform the second sub-parameter matrix using a transposed matrix of the first permutation matrix and a fourth permutation matrix;

[0023] For the third sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the third sub-parameter matrix, the third sub-parameter matrix is ​​transformed using the transposed matrix of the first permutation matrix and the fourth permutation matrix.

[0024] In yet another possible implementation, the method further includes:

[0025] Determine a first position matrix and a second position matrix corresponding to different position indexes in the sequence;

[0026] For each position index, transform the first position matrix and the second position matrix using the third permutation matrix and the confusion matrix to obtain transformed first position matrix and second position matrix;

[0027] The transformed first position matrix and the second position matrix corresponding to different position indexes are deployed to the inference program on the user device side, and the inference program is used to call the model parameters of the target neural network model for inference calculation.

[0028] In yet another possible implementation, if the target neural network model is a second type neural network model, the second parameter matrix in the target neural network model includes a fourth sub-parameter matrix and a fifth sub-parameter matrix;

[0029] The step of performing parameter transformation on the model parameters based on the transformation function corresponding to the model parameters in the target layer to obtain the transformed model parameters includes:

[0030] For a fourth sub-parameter matrix in the attention layer, based on a third transformation function corresponding to the fourth sub-parameter matrix, the fourth sub-parameter matrix is ​​transformed using a random number, a second permutation matrix, and a second reversible matrix;

[0031] For the fifth sub-parameter matrix in the attention layer, based on the third transformation function corresponding to the fifth sub-parameter matrix, the fifth sub-parameter matrix is ​​transformed using random numbers, a second permutation matrix and an inverse matrix of the second reversible matrix.

[0032] In another possible implementation, determining the target neural network model to be protected based on the model protection request includes:

[0033] Determine model requirement information based on the model protection request, where the model requirement information includes at least one of: service category information that needs to be processed by the model and computing capability information of a user device on which the model is to be deployed;

[0034] Based on at least one of the business category information and the computing capability information, a target neural network model to be protected is determined.

[0035] On the other hand, the present application also provides a model data processing method, applied to a user device, comprising:

[0036] Obtaining a model parameter set of a target neural network model sent by a model provider, the model parameter set including transformed model parameters in at least one target layer to be protected in the target neural network model, the transformed model parameters in the target layer being obtained by performing parameter transformation on the model parameters based on a transformation function corresponding to the model parameters;

[0037] The set of model parameters is deployed in the user equipment.

[0038] In yet another possible implementation, obtaining a model parameter set of a target neural network model sent by a model provider includes:

[0039] Determining model requirement information of the user equipment, the model requirement information comprising: at least one of service category information that the user equipment needs to process using the model and computing capability information of the user equipment;

[0040] Sending a model protection request to a model provider, wherein the model protection request includes the model requirement information;

[0041] Obtain a model parameter set of the target neural network model returned by the model provider in response to the model protection request.

[0042] In another aspect, the present application also provides an electronic device, including a processor and a memory;

[0043] The processor is used to execute the model data processing method as described in any one of the above;

[0044] The memory is used to store programs required for the processor to perform operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0046] Figure 1 A flow chart of a model data processing method provided in this application;

[0047] Figure 2 An example diagram of model parameters deployed on user equipment side in the direction of model provision in this application;

[0048] Figure 3 A schematic diagram of another flow chart of the model data processing method provided in this application;

[0049] Figure 4 A schematic diagram of another flow chart of the model data processing method provided in this application;

[0050] Figure 5 Another example diagram of deploying model parameters on a user device side in the direction of the model provider in this application;

[0051] Figure 6 A flowchart of another model data processing method provided by the present application;

[0052] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0053] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0054] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0055] like Figure 1 , showing a flow chart of the model data processing method provided in an embodiment of the present application. The method of this embodiment can be applied to a model provider, which can be an electronic device that provides a model, such as a server, a cloud platform for providing a model, or a device node in other systems, etc., without limitation.

[0056] The method of this embodiment may include the following steps S101 to S104:

[0057] S101, based on the model protection request, determining the target neural network model to be protected.

[0058] Among them, the model protection request is used to request protection of the neural network model.

[0059] In this application, the neural network model that needs to be protected based on the model protection request is called the target neural network model. The target neural network model includes multiple layers. The target neural network model is a hierarchical organizational structure, and different layers have different functions. Based on the specific type of the target neural network model, the specific composition structure of the target neural network model is also different.

[0060] For example, the target neural network model may include an embedding layer, an attention layer (also called an attention module), a feedforward layer (also called a feedforward module), and an output layer. Each layer may be set to only one or more. For example, the target neural network model may include multiple attention layers and multiple feedforward layers.

[0061] In the present application, there are multiple possibilities for determining the specific implementation of the target neural network model.

[0062] For example, in one possible implementation, considering that the user devices that initiate the model protection request are different, the requirements for the neural network model to be protected will also be different. Therefore, the target neural network model to be protected can be determined based on the user device that initiates the model protection request.

[0063] In another possible implementation, in order to more reasonably determine the model suitable for the user device, the present application can also determine the model requirement information based on the model protection request. The model requirement information includes at least one of the business category information that needs to be processed by the model and the computing power information of the user device to be deployed with the model. Accordingly, the target neural network model to be protected can be determined based on at least one of the business category information and the computing power information.

[0064] Among them, the service category information is used to characterize the category of services that need to be processed by the neural network model. For example, the categories of services can be divided into different categories such as information classification, information prediction, and question-and-answer response. Based on the service category information, the application scenarios in which the user device needs to use the neural network model can be determined, so that the target neural network model suitable for the user device can be determined.

[0065] Among them, the computing capability information of the user device can reflect the hardware performance of the user device side, as well as which neural network models the user device can support, etc. Based on this, the target neural network model suitable for deployment on the user device side can be determined based on the computing capability information of the user device.

[0066] There are also multiple possibilities for determining the model requirement information based on the model protection request. For example, the model protection request may include the model requirement information, based on which the model requirement information in the model protection request may be obtained. For another example, the model protection request may also include indication information that can characterize the model requirement information, based on which the model requirement information may be determined. For example, the indication information may be device information such as the type and model of the user device, or information used to ensure the function or role of the model, etc., without limitation.

[0067] S102, determining at least one target layer to be protected among the multiple layers and a transformation function corresponding to each model parameter in the target layer.

[0068] The target layer to be protected is a layer in the target neural network model whose model parameters need to be protected.

[0069] In the present application, depending on the different target neural network models and the actual protection requirements, some layers of the target neural network model can be determined as target layers, or each layer of the target neural network model can be determined as a target layer to be protected.

[0070] The transformation function corresponding to the model parameter is used to transform the model parameter to avoid directly exposing the model parameter. The transformation functions corresponding to the model parameters in different layers may be different, and the transformation functions corresponding to different model parameters in the same layer may also be different.

[0071] S103, performing parameter transformation on the model parameters based on the transformation function corresponding to the model parameters in the target layer to obtain transformed model parameters.

[0072] It can be understood that for each model parameter in each target layer, the transformation process of the model parameter can be completed in step S103 to obtain the transformed model parameter, wherein the parameter value of the transformed model parameter is different from that of the model parameter before the transformation.

[0073] Since the transformation function of the model parameters can indicate the transformation coefficients and transformation rules used to transform the model parameters, based on this, the model parameters can be transformed according to the transformation function to change the parameter values ​​of the model parameters. Among them, for different model parameters, since the transformation functions of the model parameters may be different, the specific implementation process of parameter transformation of the model parameters may also be different.

[0074] S104, deploying the transformed model parameters in each target layer to the user equipment end.

[0075] For example, in actual applications, the model provider can deploy the model parameter set of the target neural network model to the user device, and the model parameter set includes the transformed model parameters in each target layer of the target neural network model. Of course, if the target neural network model also includes other layers that do not belong to the target layer, the model parameter set can also include the untransformed model parameters in the unprotected layer.

[0076] In the present application, there is no restriction on the specific deployment method of the transformed model parameters in each target layer on the user equipment side.

[0077] It can be seen that in the present application, the target neural network model that needs to be protected can be determined based on the model protection request, so that the target neural network model that needs to be deployed to the user device side can be reasonably determined based on the actual needs of the user device side. Moreover, after determining the target layer to be protected in the target neural network model and the transformation function corresponding to each model parameter in the target layer, the present application will perform parameter transformation on the model parameters based on the transformation function of the model parameters in the target layer, and then deploy the transformed model parameters in the target layer to the user device side, which can reduce the risk of leakage of the model parameters that need to be protected in the target neural network model, thereby achieving a more reasonable deployment of the neural network model to the user device side.

[0078] In the present application, there are also multiple possible implementations for determining the target layer to be protected in the target neural network model and the transformation function corresponding to the model parameters in the target layer.

[0079] For example, in one implementation, the layer names of the target layers to be protected in different neural network models and the transformation functions corresponding to the model parameters in each target layer can be pre-configured. On this basis, after the target neural network model is determined, the pre-configured target layers to be protected in multiple layers of the target neural network model can be determined, and the transformation functions corresponding to the different model parameters in each pre-configured target layer can be obtained.

[0080] In another implementation, considering that the model composition structure of the same type of neural network model and the functions in each layer of the model are also similar, therefore, for neural network models of the same model category, the target layer to be protected and the transformation function of the model parameters in the target layer can be the same. Based on this, the present application can also determine at least one target layer to be protected in the target neural network model and the transformation function corresponding to each model parameter in each target layer in combination with the model category of the target neural network model.

[0081] Among them, different types of neural network models can pre-configure the model protection strategy of the neural network model of the model type, based on which the model protection strategy can include information of the target layer to be protected and transformation functions of different model parameters in the target layer. Alternatively, the model protection strategy can include information of the target layer to be protected and transformation rules of the model parameters in the target layer, and the transformation function of the model parameters is determined based on the transformation rules of the model parameters.

[0082] Combining the categories of different neural network models to determine the target layer to be protected and the transformation function of the model parameters in the target layer can combine the model characteristics of different categories of neural network models to more reasonably determine the target layer to be protected and the transformation function of the model parameters.

[0083] It can be understood that in the neural network model based on the attention mechanism, the attention layer is a relatively important and critical layer. Therefore, under the premise that the target neural network model includes the attention layer, the target layer to be protected in the target neural network model at least includes the attention layer.

[0084] The following is an explanation based on different model categories of the target neural network model:

[0085] When the target neural network model is a first type of neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and the first transformation function corresponding to each first parameter matrix in the attention layer and the second transformation function corresponding to each first normalization parameter in the attention layer are determined.

[0086] Among them, the first transformation function includes a first permutation matrix and a set interference coefficient. The first permutation matrix is ​​used to permute the first parameter matrix to change the position of each parameter in the first parameter matrix. The interference coefficient is used to transform the parameter values ​​corresponding to the first parameter matrix to increase the difficulty of cracking the first parameter matrix. For example, the interference coefficient may include at least one of a confusion matrix, a randomly generated random number, a reversible matrix, and an inverse matrix corresponding to the reversible matrix. Of course, the interference coefficient may also be a permutation matrix other than the first permutation matrix.

[0087] The second transformation function includes a first permutation matrix.

[0088] When the target neural network model is a second type of neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and the third transformation function corresponding to each second parameter matrix in the attention layer and the fourth transformation function corresponding to each second normalization parameter in the attention layer are determined.

[0089] In the present application, the third transformation function and the fourth transformation function include a second permutation matrix.

[0090] For the sake of distinction, when the target neural network model is a first type neural network model, the parameter matrix of the attention layer in the target neural network model is called a first parameter matrix, and the normalization parameter in the attention layer of the target neural network model is called a first normalization parameter. Correspondingly, when the target neural network model is a second type neural network model, the parameter matrix in the attention layer of the target neural network model is called a second parameter matrix, and the normalization parameter in the attention layer is called a second normalization parameter.

[0091] The following describes different implementation methods of the transformation function for determining the target layer and model parameters in the target layer in combination with possible model categories of the target neural network model.

[0092] like Figure 2 , which is another flow chart of the model data processing method provided in the embodiment of the present application. This embodiment is applied to the model provider. The method of this embodiment may include:

[0093] S201, based on the model protection request, determining the target neural network model to be protected.

[0094] The target neural network model includes multiple layers.

[0095] For this step, please refer to the related introduction of the previous embodiment, which will not be repeated here.

[0096] S202, in response to the target neural network model being a first type of neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and a first transformation function corresponding to each first parameter matrix in the attention layer and a second transformation function corresponding to each first normalization parameter in the attention layer are determined.

[0097] In the case where the target neural network model is a first type of neural network model, the first parameter matrix in the target neural network model may include: a first sub-parameter matrix, a second sub-parameter matrix and a third sub-parameter matrix, wherein the first sub-parameter matrix, the second sub-parameter matrix and the third sub-parameter matrix belong to three types of parameter matrices in the target neural network model that need to be transformed using different transformation functions. Therefore, each of the three sub-parameter matrices in the target neural network model may be one or more.

[0098] In the present application, the first transformation function includes a first permutation matrix and a set interference coefficient, and the set interference coefficient can be at least one of a random number, a reversible matrix, or an inverse matrix of a reversible matrix. For different sub-parameter matrices, the interference coefficient in the first transformation function will also be different.

[0099] The interference coefficient in the first transformation function corresponding to the first sub-parameter matrix is ​​a random number.

[0100] The interference coefficients in the first transformation function corresponding to the second sub-parameter matrix are random numbers and a first reversible matrix.

[0101] The interference coefficients in the first transformation function corresponding to the third sub-parameter matrix are random numbers and the inverse matrix of the first reversible matrix.

[0102] Among them, the random numbers contained in the first transformation functions corresponding to different sub-parameter matrices can be different random numbers. In order to ensure that the output results of the target neural network model are not affected after the transformation of each model parameter, usually, the random numbers in the first transformation functions of different sub-parameter matrices are different, so as to combine the mutual relationship between the functions in each layer of the target neural network model to achieve the purpose of offsetting the influence of random numbers.

[0103] The second transformation function includes the first permutation matrix. Of course, the second transformation function may also include random numbers.

[0104] S203, for the first sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the first sub-parameter matrix, the first sub-parameter matrix is ​​transformed using random numbers and the transposed matrix of the first permutation matrix to obtain a transformed first sub-parameter matrix.

[0105] The random number is randomly generated data, and the random number is a random number included in the first transformation function corresponding to the first sub-parameter matrix. If there are multiple first sub-parameter matrices, the random numbers in the first transformation functions corresponding to different sub-parameter matrices may be different.

[0106] For example, the first transformation function may be a random number, a transposed matrix of the first permutation matrix, and the first sub-parameter matrix multiplied in sequence. Then, based on the first transformation matrix, the first sub-parameter matrix may be left-multiplied by the random number and the transposed matrix of the first permutation matrix to obtain the transformed first sub-parameter matrix.

[0107] S204, for the second sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the second sub-parameter matrix, the second sub-parameter matrix is ​​transformed using random numbers, the transposed matrix of the first permutation matrix and the first reversible matrix to obtain a transformed second sub-parameter matrix.

[0108] The random number in this step is a random number in the first transformation function corresponding to the second sub-parameter matrix, and is different from the random number in the first transformation function corresponding to the first sub-parameter matrix and the third sub-parameter matrix.

[0109] S205, for the third sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the third sub-parameter matrix, the third sub-parameter matrix is ​​transformed using random numbers, the first permutation matrix and the inverse matrix corresponding to the first reversible matrix to obtain a transformed third sub-parameter matrix.

[0110] Among them, using random numbers to transform each sub-parameter matrix can change the parameter value of each parameter in the sub-parameter matrix, thereby increasing the difficulty of cracking the sub-parameter matrix. In this embodiment, the random numbers used to transform different sub-parameter matrices can be different. Specifically, the random numbers used by each sub-parameter matrix can be determined in combination with the functional relationship of each sub-parameter matrix in the attention layer, so that the function operation of each sub-parameter matrix in the attention layer can offset the influence of each random number on the output result of the attention layer or the target neural network.

[0111] It is understandable that if each sub-parameter matrix in the attention layer is transformed based only on the permutation matrix, then only the position of each parameter in each sub-parameter matrix is ​​changed, making the sub-parameter matrix less difficult to crack, but still less secure. However, the present application adds random numbers to the transformation function of each sub-parameter matrix, thereby being able to change the parameter values ​​of each parameter in the sub-parameter matrix, making it more difficult to crack the sub-parameter matrix, and improving the security of the sub-parameter matrix.

[0112] In the present application, the second sub-parameter matrix and the third sub-parameter matrix are two sub-parameter matrices that have multiplication operations in the attention layer. In this case, the present application uses a reversible matrix and an inverse matrix of the reversible matrix to transform the two sub-parameter matrices respectively, and then multiplies the two transformed sub-parameter matrices in the attention layer to achieve the effect of multiplying the reversible matrix with its inverse matrix, and the reversible matrix and its inverse matrix are multiplied into a unit matrix, and any matrix multiplied with the unit matrix is ​​still equal to the matrix itself, so that the sub-parameter matrix will not affect the output result of the attention layer after being transformed.

[0113] As for the second sub-parameter matrix and the third sub-parameter matrix, since a reversible matrix or an inverse matrix is ​​added during the transformation process, the difficulty of cracking the transformed second sub-parameter matrix and the third sub-parameter matrix can be further increased, thereby improving the security of the two sub-parameter matrices.

[0114] S206: For the first normalized parameter in the attention layer, based on the second transformation function corresponding to the first normalized parameter, the first normalized parameter is transformed using the first permutation matrix to obtain the transformed first normalized parameter.

[0115] In particular, in order to improve the security of the normalized parameters, the second transformation function may further include a random number. Based on this, the first normalized parameters may be transformed based on the second transformation function using the random number and the first permutation matrix.

[0116] It should be noted that the order of the above steps S203 to S206 is not limited to Figure 2 As shown, in actual applications, these steps can be interchanged in order or executed simultaneously without restriction.

[0117] It is understandable that, when the target neural network model is a first-class neural network model, the above is an example in which the target layer to be protected in the target neural network model includes an attention layer. In practical applications, if the target neural network model also includes a feedforward layer, the present application can also determine the feedforward layer as a target layer to be protected.

[0118] Correspondingly, when the target neural network model is a first type of neural network model, for the first feedforward parameter matrix in the feedforward layer, the first feedforward transformation function corresponding to each first feedforward parameter matrix can be determined. The first feedforward transformation function can include a first permutation matrix and a random number. On this basis, the process of performing parameter transformation on the first feedforward parameter matrix based on the first feedforward transformation function can be similar to the transformation process of the first sub-parameter matrix, which will not be repeated.

[0119] Furthermore, in order to improve the security of the first feedforward parameter matrix, the first feedforward transformation function includes a random number, a first permutation matrix and a first auxiliary permutation matrix. Among them, the random numbers included in the first feedforward transformation function corresponding to different first feedforward parameter matrices are different. On this basis, based on the first feedforward transformation function corresponding to the first feedforward parameter matrix, the first feedforward parameter matrix can be transformed using random numbers, the first permutation matrix and the first auxiliary permutation matrix to obtain the transformed first feedforward parameter matrix. On this basis, in the transformed first feedforward parameter matrix, not only the positions of the parameters change, but also the values ​​of the parameters change, thereby reducing the risk of the first feedforward parameter matrix being cracked.

[0120] Among them, if the feedforward layer includes multiple first feedforward parameter matrices, the first feedforward transformation function corresponding to different first feedforward parameter matrices can be to transpose at least one of the first permutation matrix and the first auxiliary permutation matrix. Accordingly, based on the first feedforward transformation function, parameter transformation of the first feedforward parameter matrix is ​​actually multiplying the first feedforward parameter matrix with the transpose of at least one of the first permutation matrix and the first auxiliary permutation matrix.

[0121] In order to further improve the security of the model parameters in the target neural network model, the present application can also determine all layers in the target neural network model as target layers to be protected. Based on this, the target layer can also include the input layer and output layer of the target neural network model.

[0122] Among them, for the first input layer parameter in the input layer, the first input transformation function corresponding to the first input layer parameter can be determined. The first input transformation function can include the first permutation matrix. Further, in order to improve the security of the first input layer parameter, the first input transformation function can also include a random number. Accordingly, based on the first input transformation function corresponding to the first input layer parameter, the first input layer parameter is transformed using the random number in the first input transformation function and the first permutation matrix to obtain the transformed first input layer parameter.

[0123] Similarly, for the first output layer parameter in the output layer, the first output transformation function corresponding to the first output layer parameter can be determined. The first output transformation function can include the first permutation matrix. Further, in order to improve the security of the first output layer parameter, the first output transformation function can also include a random number. Accordingly, based on the first output transformation function corresponding to the first output layer parameter, the first output layer parameter is transformed based on the random number in the first output transformation function and the first permutation matrix to obtain the transformed first output layer parameter.

[0124] S207, in response to the target neural network model being a second type of neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and the third transformation function corresponding to each second parameter matrix in the attention layer and the fourth transformation function corresponding to each second normalization parameter in the attention layer are determined.

[0125] It can be understood that when the target neural network model is a second type neural network model, although the target neural network model also includes an attention layer, the structure of the attention layer is different from that of the attention layer in the target neural network model belonging to the first type neural network model, and accordingly, the parameter matrix and normalization parameters contained in the attention layer will also be different.

[0126] In the case where the target neural network model is a second type neural network model, the second parameter matrix in the target neural network model includes a fourth sub-parameter matrix and a fifth sub-parameter matrix. The fourth sub-parameter matrix and the fifth sub-parameter matrix are two sub-parameter matrices that have a multiplication operation in the attention layer. In practical applications, the second parameter matrix may include multiple pairs of interrelated fourth sub-parameter matrices and fifth sub-parameter matrices.

[0127] The third transformation function corresponding to the fourth sub-parameter matrix may include a random number, a second permutation matrix, and a second reversible matrix. The third transformation function corresponding to the fifth sub-parameter matrix may include a random number, a second permutation matrix, and an inverse matrix corresponding to the second reversible matrix. The random number in the third transformation function corresponding to the fourth sub-parameter matrix and the third transformation function corresponding to the fifth sub-parameter matrix may be different.

[0128] The fourth transformation function at least includes the second transposed matrix. Optionally, in order to further improve the security of the second normalization parameter, the fourth transformation function may also include a random number, and the random number in the fourth transformation function is different from the random number in the third transformation function.

[0129] S208, for the fourth sub-parameter matrix in the attention layer, based on the third transformation function corresponding to the fourth sub-parameter matrix, the fourth sub-parameter matrix is ​​transformed using random numbers, a second permutation matrix and a second reversible matrix to obtain a transformed fourth sub-parameter matrix.

[0130] Among them, the third transformation function corresponding to the fourth sub-parameter matrix may include random numbers and functional operations between the second permutation matrix and the fourth sub-parameter matrix. Based on this, the fourth sub-parameter matrix can be transformed according to the third transformation function using random numbers, the second permutation matrix and the second reversible matrix.

[0131] For example, the fourth sub-parameter matrix is ​​left-multiplied by a random number, a second permutation matrix (or the transpose of the second permutation matrix), and a second reversible matrix.

[0132] S209, for the fifth sub-parameter matrix in the attention layer, based on the third transformation function corresponding to the fifth sub-parameter matrix, the fifth sub-parameter matrix is ​​transformed using random numbers, a second permutation matrix and an inverse matrix of the second reversible matrix to obtain a transformed sub-parameter matrix.

[0133] Similar to step S208, the third transformation function corresponding to the fifth sub-parameter matrix may include function operations of random numbers, the second permutation matrix, and the inverse matrix of the second reversible matrix. Based on this, the fifth sub-parameter matrix can be transformed in parameters in combination with the third transformation function corresponding to the fifth sub-transformation matrix.

[0134] S210, for the second normalized parameter in the attention layer, based on the fourth transformation function corresponding to the second normalized parameter, the second normalized parameter is transformed using a second permutation matrix to obtain a transformed second normalized parameter.

[0135] In particular, in order to improve the security of the second normalization parameter, the fourth transformation function may further include a random number. Based on this, the second normalization parameter may be transformed based on the fourth transformation function using the random number and the second permutation matrix.

[0136] It should be noted that the order of the above steps S203 to S206 is not limited to Figure 2 As shown, in actual applications, these steps can be interchanged in order or executed simultaneously without restriction.

[0137] It can be understood that when the target neural network model is a second type neural network model, steps S207 to S209 are an example of an implementation method for parameter transformation of the fourth sub-parameter matrix and the fifth sub-parameter matrix in the attention layer, and this implementation method is similar to the transformation process of the sub-parameter matrix in the first type neural network model, and at least one of a random number, a reversible matrix and an inverse matrix of a reversible matrix is ​​also used, so it is also possible to increase the difficulty of cracking the fourth sub-parameter matrix and the fifth sub-parameter matrix. For details, please refer to the previous introduction to the effect of using random numbers and reversible matrices (or inverse matrices), which will not be repeated here.

[0138] However, it can be understood that, in steps S207 and S209, if the third transformation function only includes the second permutation matrix, then the fourth sub-parameter matrix and the fifth sub-parameter matrix can be transformed only based on the second permutation matrix, and this implementation is also applicable to the present embodiment without limitation.

[0139] It can be understood that, in the case where the target neural network model is a second type of neural network model, if the target neural network model also includes a feedforward layer, then the target layer to be protected in the target neural network model may also include a feedforward layer. For the second type of neural network model, the feedforward parameter matrix in the feedforward layer is different from the first feedforward parameter matrix. For ease of distinction, under the premise that the target neural network is a second type of neural network model, the feedforward parameter matrix in the feedforward layer is called the second feedforward parameter matrix.

[0140] For the second feedforward parameter matrix in the feedforward layer, a second feedforward transformation function corresponding to each second feedforward parameter matrix can be determined. The second feedforward transformation function may include a second permutation matrix and a random number. On this basis, based on the second feedforward transformation function, the second feedforward parameter matrix is ​​parameter transformed using the second permutation matrix and the random number.

[0141] Furthermore, in order to improve the security of the second feedforward parameter matrix, the second feedforward transformation function includes a random number, a second permutation matrix and a second auxiliary permutation matrix. Among them, the random numbers included in the second feedforward transformation function corresponding to different second feedforward parameter matrices are different. On this basis, based on the second feedforward transformation function corresponding to the second feedforward parameter matrix, the second feedforward parameter matrix can be transformed using random numbers, a second permutation matrix and a second auxiliary permutation matrix to obtain a transformed second feedforward parameter matrix. On this basis, in the transformed second feedforward parameter matrix, not only the positions of the parameters change, but also the values ​​of the parameters change, thereby reducing the risk of the second feedforward parameter matrix being cracked.

[0142] Among them, if the feedforward layer includes multiple second feedforward parameter matrices, the second feedforward transformation function corresponding to different second feedforward parameter matrices can be to transpose at least one of the second permutation matrix and the second auxiliary permutation matrix. Accordingly, based on the second feedforward transformation function, parameter transformation of the second feedforward parameter matrix is ​​actually multiplying the second feedforward parameter matrix with the transpose of at least one of the second permutation matrix and the second auxiliary permutation matrix.

[0143] In order to further improve the security of the model parameters in the target neural network model, the present application can also determine all layers in the target neural network model as target layers to be protected. Based on this, the target layer can also include the input layer and output layer of the target neural network model.

[0144] In the case where the target neural network model is a second type of neural network model, for the second input layer parameters in the input layer, a second input transformation function corresponding to the second input layer parameters can be determined. The second input transformation function can include the second permutation matrix. Further, in order to improve the security of the second input layer parameters, the second input transformation function can also include a random number. Accordingly, based on the second input transformation function corresponding to the second input layer parameters, the second input layer parameters are transformed using the random number in the second input transformation function and the second permutation matrix to obtain the transformed second input layer parameters.

[0145] Similarly, for the second output layer parameters in the output layer, a second output transformation function corresponding to the second output layer parameters can be determined. The second output transformation function can include the second permutation matrix. Further, in order to improve the security of the second output layer parameters, the second output transformation function can also include a random number. Accordingly, based on the second output transformation function corresponding to the second output layer parameters, the output layer parameters are transformed using the random number in the second output transformation function and the second permutation matrix to obtain the transformed output layer parameters.

[0146] S211, deploy the transformed model parameters in each target layer including the attention layer to the user device end.

[0147] For example, if the target layer also includes at least one of the feedforward layer, the input layer, and the output layer, the transformed model parameters in these layers can also be deployed to the user device. If the target layer only includes the attention layer, the transformed model parameters in the attention layer and the untransformed model parameters in other layers can be deployed to the user device.

[0148] like Figure 3 As shown in FIG. 1 , it shows an example diagram of a model provider deploying model parameters to a user device in the present application. Figure 3 It can be seen that the model provider will transform the model parameters in the target neural network model, and deploy the new model parameters obtained after the transformation to the user device side, so that the user device side can call the deployed transformed model parameters based on the input content to determine the output result.

[0149] For ease of understanding Figure 2 The embodiments and their benefits are described below by taking a target neural network model including an input layer, an attention layer, a feedforward layer and an output layer as an example, and combining it with an application example.

[0150] For example, when the neural network model has an input layer, an attention layer, a feedforward layer, and an output layer, the neural network model can be divided into two categories. It is assumed that the first type of neural network model is the LLaMA (Large Language Model Meta AI) model, and the second type of neural network model is the Transformer model.

[0151] First, the conversion process of model parameters in the Transformer model is introduced in combination with an implementation method.

[0152] For the Transformer model, the following permutation matrix and random numbers can be pre-built:

[0153] Randomly generate a permutation matrix π∈{0,1} d×d , generate a list L containing N permutation matrices P =[π1,…,π N |π i ∈{0,1} d×d ], N is the total number of attention layers (or feedforward layers), and i is a natural number from 1 to N. Among them, d is the feature dimension corresponding to the input layer, which is also the feature dimension of the attention layer. The meanings of N and d apply not only to the Transformer model, but also to the input layer and attention layer in the LLaMA model. For other parameters that appear in the subsequent LLaMA model transformation process, if they are the same as the corresponding parameters in the Transformer model, then the meanings of the parameters are also the same and will not be repeated one by one. π∈{0,1} d×d Indicates that the permutation matrix π is a d×d matrix, in which each element is either 0 or 1. i It is similar and I will not go into details.

[0154] Randomly generate two lists L, each containing N reversible matrices B =[B1,…,B N |B i ∈R d×d ] and L C =[C1,…,C N |C i ∈R d×d ] Calculate the inverse matrix of each reversible matrix in the two reversible matrix lists respectively, and get two inverse matrix lists respectively. and Among them, R d×d Represents a d×d matrix.

[0155] Generate three random number lists L a =[a1,…,a N+1 |ai ∈R],L d =[d1,…,d N |d i ∈R],L e =[e1,…,e N |e i ∈R]. Where R represents the field of real numbers.

[0156] The permutation matrices and random numbers generated above can form a parameter set, namely, the parameter set Π1={π,L P ,L B ,L C ,L Binv ,L Cinv ,L a ,L d ,L e}.

[0157] On this basis, the parameter conversion of each model parameter of each layer in the Transformer model is explained. The following is the conversion function corresponding to the model parameters of each layer:

[0158] For the input layer:

[0159] W e ′=a1W e π;

[0160] Among them, W e is the model parameter of the input layer, that is, the input layer parameter. e ′ is W e The input layer parameters obtained after transformation (in the Transformer model, the input layer parameters are the second input layer parameters). It can be seen that for the input layer parameters W e By left-multiplying a random number a1 and the permutation matrix π, we get the transformed input layer parameter W e ′.

[0161] For the attention layer, since there are multiple attention layers in the Transformer model, taking the i-th attention layer as an example, the transformation function corresponding to each model parameter in the attention layer is as follows:

[0162]

[0163] γ′ 1i =d i γ 1i π;

[0164] β′ 1i =d i β 1i π;

[0165] Among them, Wqi , W ki , W vi , W oi Respectively represent different parameter matrices in the i-th attention layer. Among them, W′ qi , W′ ki , W v ' i , W o ' i W qi , W ki , W vi , W oi The parameter matrix obtained after parameter transformation. Correspondingly, β 1i , β 1i is the normalization parameter in the i-th attention layer (the normalization parameter of the attention layer in the Transformer model is also the second normalization parameter mentioned above). 1i , β′ 1i They are γ 1i , β 1i The parameter matrix obtained after transformation.

[0166] Among them, W qi and W ki It can be a set of corresponding fourth sub-parameter matrix and fifth sub-parameter matrix. qi The corresponding transformation function shows that using the random number a i The reciprocal of Permutation matrix π and reversible matrix B i The transpose (π T ) and the reversible matrix B i Left multiply W qi , we get the transformed parameter matrix W′ qi . And by W ki The corresponding transformation function shows that using the random number a i The reciprocal of The transpose of the permutation matrix π(π T ) and the reversible matrix B i The corresponding inverse matrix Transpose Left multiply W ki , we can get W′ ki Combined with the previous introduction to the attention layer operator, we can see that in the attention layer operator W′ ki and W′ qi There will be a multiplication operation, so that the reversible matrix B i Its inverse matrix After multiplication, it becomes an identity matrix.

[0167] Similarly, W′vi and Wv oi It can also be a set of a fourth sub-parameter matrix and a fifth sub-parameter matrix. The transformation of these two parameters can also be the above transformation function, which will not be repeated.

[0168] For γ 1i , β 1i Both of these normalized parameters can be transformed by left-multiplying a random number and a permutation matrix, which will not be described in detail.

[0169] For the feedforward layer, since there are multiple feedforward layers in the Transformer model, the i-th feedforward layer is taken as an example. The transformation function corresponding to each model parameter in the feedforward layer is as follows:

[0170] W′ 1i =e i π T W 1i π i ;

[0171]

[0172] γ′ 2i =a i+1 γ 2i π;

[0173] β′ 2i =a i+1 β 2i π;

[0174] In order to distinguish, W 1i , W 2i W′ represents the feed-forward layer parameter matrix in the i-th feed-forward layer. 1i , W′ 2i W 1i , W 2i The feedforward layer parameter matrix obtained after transformation. Similarly, γ 2i , β 2i is the normalization parameter in the i-th feed-forward layer.

[0175] It can be seen that when the target neural network model is a second type neural network model, the feedforward layer parameter matrix in the feedforward layer can use random numbers, a permutation matrix π (specifically, the transpose of the permutation matrix π) and an auxiliary permutation matrix π i For the normalization parameter in the feedforward layer, a random number a can be used i+1 And transform it with the permutation matrix π.

[0176] Among them, the transformation function of the model parameters in the output layer is as follows:

[0177]

[0178] Among them, W c is the model parameter in the output layer, that is, the output layer parameter, W′ c is the output layer parameter W c The output layer parameters obtained after transformation.

[0179] When the parameters of each model in the Transformer model are transformed, the function operation of each model parameter in each layer and the function operation relationship of the model parameters between different layers are fully considered, so that after each model parameter in the Transformer model is replaced with the transformed model parameters, the output result of the Transformer model can be unaffected. In order to prove that the output result of the Transformer model can remain unchanged when the transformed model parameters are deployed in the Transformer model, it can be deduced and proved in combination with the functions in each layer of the Transformer model.

[0180] Among them, considering that the attention layer and embedding layer in the Transformer model can have multiple layers, in order to simplify the derivation, only the input of the attention layer is used as the output result of the input layer as an example. Correspondingly, after each model parameter in the Transformer model is replaced with the transformed model parameter, the input information of the Transformer model is processed by each layer in turn. The function processing involved is as follows:

[0181] In the input layer: x′1 = T input W′ e =a1x1π;

[0182] Among them, x1 is the output representation of the input layer in the Transformer model when the input layer parameters of the input layer are not transformed, and correspondingly, x′1 is the output representation of the input layer when the input layer parameters are transformed input layer parameters.

[0183] In the attention layer:

[0184]

[0185] v′ i =LayerNorm(u′ i +x′ i ; γ′ 1i ,β′ 1i )

[0186] =LayerNorm(a i u i π+a i x i π;di γ1iπ,d i β 1i π)

[0187] =d i LayerNorm((u i +x i )π;γ 1i π,β 1i π)=d i v i π;

[0188] Among them, Q i , K i and V i are the query vector, key-value vector, and value vector in the i-th attention layer, respectively. Q′ i , K′ i and V′ i They represent the query vector, key-value vector, and value vector after the model parameters in the i-th attention layer are replaced by the transformed model parameters. i and v i is the intermediate output and final output of the i-th attention layer, u′ i and v′ i They are the intermediate output and output results obtained when the transformed model parameters are deployed in the i-th attention layer. SoftMax() is a normalization function. where k is a constant. LayerNorm() is another normalization function, where M is a parameter matrix with fixed values ​​for each parameter.

[0189] In the feed-forward layer:

[0190]

[0191] y′ i =LayerNorm(z′ i +v′ i ; γ′ 2i ,β′ 2i )

[0192] =LayerNorm(d i z i π+d i v i π;a i+1 γ 2i π,a i+1 β 2i π)

[0193] =a i+1 LayerNorm((z i+v i )π;γ 2i π,β 2i π)=a i+1 y i π;

[0194] Among them, z i and i Respectively represent the intermediate output result of the i-th feedforward layer and the output result of the i-th feedforward layer, z′ i and y′ i They are respectively the intermediate output result and the output result of the i-th feedforward layer when the model parameters in the i-th feedforward layer are replaced by the transformed model parameters. ReLU() represents the linear rectification function.

[0195] Output layer:

[0196]

[0197] Among them, output represents the output result of the output layer in the Transformer model, that is, the output result of the Transformer model, and output′ is the output result of the Transformer model after each model parameter in the Transformer model is replaced by the transformed model parameters.

[0198] It can be seen that after the model parameters in the Transformer model are replaced by the transformed model parameters, the final output result of the output layer of the Transformer model is still the original output, which is consistent with the output result before the model parameters in the previous Transformer model are transformed.

[0199] The following describes the target neural network model being the first type of neural network model, that is, the target neural network model being the component structure of each layer in the LLaMA model and the process of transforming the model parameters in each layer.

[0200] The following is an introduction to the process of transforming the model parameters in the LLaMA model. Since the structure of some layers in the LLaMA model and the Transformer model are similar, the model parameters involved are also partially the same. The same characters are used to represent the same model parameters. The specific meanings can be referred to the corresponding introduction in the previous Transformer model, and no further explanation is given.

[0201] For the LLaMA model, the following permutation matrix and random numbers can be pre-constructed:

[0202] Randomly generate a permutation matrix π∈{0,1} d×d, and L, a list of permutation matrices containing N permutation matrices P =[π1,…,π N |π i ∈{0,1} d×d ];

[0203] Randomly generate a reversible matrix list L containing N reversible matrices C =[C1,…,C N |C i ∈R d×d ], calculate the inverse matrix of each reversible matrix in the reversible matrix list, and get the inverse matrix list

[0204] Generate four random number lists: L a =[a1,…,a N |a i ∈R],L b =[b1,…,b N |b i ∈R],L d =[d1,…,d N |d i ∈R],L e =[e1,…,e N |e i ∈R]. Randomly generate two random numbers r, f∈R.

[0205] Thus, we can get a parameter set consisting of random numbers, reversible matrices and inverse matrices. The parameter set Π2={π,L P ,,L C ,,L Cinv ,L a ,L b ,L d ,L e ,r,f}.

[0206] Based on the above, the conversion functions corresponding to the model parameters of each layer in the LLaMA model and the conversion process of the model parameters are explained:

[0207] Among them, for the input layer:

[0208] W′ e =rW e π;

[0209] W′ e is the original input layer parameter W in the input layer e The input layer parameters obtained after transformation (in the LLaMA model, the input layer parameters are the first input layer parameters). It can be seen that in the LLaMA model, the input layer parameters W eBy multiplying a random number r on the left and a permutation matrix π on the right, we get the transformed input layer parameter W′ e .

[0210] For the i-th attention layer:

[0211] γ′ 1i =a i γ 1i π;

[0212]

[0213] γ′ 2i =d i γ 2i π;

[0214] Among them, in the LLaMA model, W qi and W ki can be regarded as the first sub-parameter matrix in the i-th attention layer. By getting W′ qi and W′ ki From the transformation function, we can see that the parameter matrix W in the i-th attention layer qi and W ki We can get W′ by left-multiplying the random number and the transposed matrix of the permutation matrix respectively. qi and W′ ki , where W qi and W ki The random numbers used for transformation are different, such as W qi Based on random numbers and the transpose matrix π of the permutation matrix π T Perform parameter transformation, and W′ ki The random number used for the transformation is

[0215] Among them, W vi In fact, it is the second sub-parameter matrix in the i-th attention layer, combined with W vi From the transformation function, we can see that using random numbers The transpose matrix π of the permutation matrix π T and a reversible matrix C i To W vi Transform and get W′ vi .

[0216] W oi In fact, it is the third sub-parameter matrix in the i-th attention layer, combined with W oi From the transformation function of W oi Left multiply random number r, reversible matrix C i The inverse matrix And multiply it by the permutation matrix π on the right, we can get oi The transformed model parameters W′ oi .

[0217] γ′ 1i and γ′ 2i are the normalization parameters γ in the i-th attention layer. 1i and γ 2i , after being compared with a random number (such as a random number a i or i ) is multiplied by the permutation matrix π to obtain the transformed normalized parameters.

[0218] The transformation function of each model function in the i-th feedforward layer can be as follows:

[0219]

[0220] Among them, W 1i , W 2i and W 3i are the three different parameter matrices in the i-th feed-forward layer in the LLaMA model, W′ 1i , W′ 2i and W′ 3i They are the representations of three different parameter matrices in the i-th feed-forward layer after the model parameters in the LLaMA model are replaced by the transformed model parameters. 1i , W 2i and W 3i From the conversion function, we can know that for the first type of neural network model, namely the LLaMA model, we can use random numbers, permutation matrices (specifically, the permutation matrix in the conversion function can be the transpose of the permutation matrix) and auxiliary matrix π i (The conversion function can be specifically the auxiliary matrix π i The feedforward parameter matrix (i.e., the first feedforward parameter matrix) is parameter transformed.

[0221] Output Module:

[0222] γ′3=fγ3π;

[0223]

[0224] Among them, γ′3 is the output layer parameter obtained by parameter transformation of the output layer parameter γ3. c From the corresponding transformation function, it can be seen that for the LLaMA model belonging to the first type of neural network model, its output layer parameters can be transformed using random numbers and permutation matrices.

[0225] When the parameters of each model in the LLaMA model are transformed, the function operation of each model parameter in each layer and the function operation relationship of the model parameters between different layers are fully considered, so that after each model parameter in the LLaMA model is replaced with the transformed model parameters, the output result of the LLaMA model can be unaffected. In order to prove that the output result of the LLaMA model can remain unchanged after the transformed model parameters are deployed in the LLaMA model, it can be deduced and proved by combining the functions in each layer of the LLaMA model.

[0226] After the model parameters in the LLaMA model are replaced with the transformed model parameters, the input information of the LLaMA model is processed by each layer in turn. The function processing process involved is as follows:

[0227] In the attention layer of the LLaMA model:

[0228] in′1=T input W′ e = rx0π;

[0229] x′ i =RMSNorm(in′ i ; γ′ 1i )

[0230] =RMSNorm(rin i π;a i γ 1i π)=a i x i π;

[0231]

[0232]

[0233] v′ i =RMSNorm(u′ i +in′ i ; γ′ 2i )

[0234] =RMSNorm(ru i π+rin i π;d i γ 2i π)

[0235] =d i LayerNorm((u i +in i )π;γ 2i π)=d i v i π;

[0236] Among them, i is the input of the i-th attention layer in the LLaMA model, and in′1 is the input of the attention layer under the premise of deploying the transformed model parameters in the i-th attention layer of the LLaMA model. i The initial transformation result of the input of the i-th attention layer, x′ i The initial transformation result of the input obtained by the attention layer, under the premise of deploying the transformed model parameters in the i-th attention layer of the LLaMA model. Q′ i , K′ i and V′ i The meaning of u′ can be found in the previous introduction and will not be repeated here. i and v′ i They are the intermediate output and final output of the ith attention layer under the premise that the transformed model parameters are deployed in the ith attention layer. RMSNorm() is the root mean square error normalization function, and PosEbd() represents the rotation position encoding function.

[0237] In the feed-forward layer of the LLaMA model:

[0238]

[0239] y′ i =u′ i +in′ i +z′ i =ru i π+rin i π+rz i π=ry i π=ru i π+rin i π+rz i π=ry i ;

[0240] Among them, z i is the intermediate output result of the i-th feed-forward layer, y i is the output of the i-th feed-forward layer. i and y′ i They are the intermediate output result and the final output of the i-th feed-forward layer when the feed-forward layer parameters in the i-th feed-forward layer are replaced by the transformed model parameters. SwiGLU() is an activation function, and sigmoid() is a neural network model activation function.

[0241] In the output layer of LLaMA:

[0242] o′=RMSNorm(y′ N; γ′3)=RMSNorm(ry N π; fγ3π) = foπ;

[0243]

[0244] o is the intermediate result of the output layer, and o′ is the output result of the output layer under the premise of deploying the transformed model parameters in the output layer.

[0245] After the model parameters of each layer of LLaMA are replaced with the transformed model parameters, the output result output′ of the output layer of the LLaMA model is the same as the output result output when the model parameters of each layer of LLaMA are not transformed, and the output result will not be changed.

[0246] From the above analysis, we can see that Figure 2 In an embodiment, for a target neural network model belonging to the first type of neural network model or the second type of neural model, after transforming each model parameter in the target neural network model, the transformed model parameters of each layer are deployed to the user device end. The user device end can perform task processing normally based on the model after the model parameters are transformed, and the output result will not be affected.

[0247] In conjunction with yet another implementation, the process of determining the target layer and the transformation function of the model parameters in the target layer is described below.

[0248] like Figure 4 , which is another flow chart of the model data processing method provided in the embodiment of the present application. This embodiment is applied to the model provider. The method of this embodiment may include:

[0249] S401, based on the model protection request, determining the target neural network model to be protected.

[0250] The target neural network model includes multiple layers.

[0251] For this step, please refer to the related introduction of the previous embodiment, which will not be repeated here.

[0252] S402, in response to the target neural network model being a first type of neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and a first transformation function corresponding to each first parameter matrix in the attention layer and a second transformation function corresponding to each first normalization parameter in the attention layer are determined.

[0253] In this embodiment, when the target neural network model is a first type neural network model, the first parameter matrix in the attention layer includes a first sub-parameter matrix, a second sub-parameter matrix and a third sub-parameter matrix.

[0254] Among them, the first transformation function corresponding to the first sub-parameter matrix includes a first permutation matrix, a third permutation matrix and a confusion matrix. At this time, the first transformation function is one or more relationship functions between the first permutation matrix, the third permutation matrix, the confusion matrix and the first parameter matrix.

[0255] The first transformation function corresponding to the second sub-parameter matrix includes a first permutation matrix and a fourth permutation matrix. The first transformation function is a conversion function between the first permutation matrix, the fourth permutation matrix and the first parameter matrix.

[0256] The first transformation function corresponding to the third sub-parameter matrix includes a first permutation matrix and a fourth permutation matrix, and the first transformation function corresponding to the third sub-parameter matrix is ​​different from the first transformation function corresponding to the second sub-parameter matrix.

[0257] The second transformation function corresponding to the first normalization parameter may include a first permutation matrix.

[0258] S403, for the first sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the first sub-parameter matrix, the first sub-parameter matrix is ​​transformed using the transpose matrix of the first permutation matrix and the third permutation matrix, and the transformed first sub-parameter matrix is ​​secondarily transformed in combination with the confusion matrix to obtain the transformed first sub-parameter matrix.

[0259] For example, the first sub-parameter matrix is ​​multiplied on the left by the transposed matrix of the first permutation matrix, and the first sub-parameter matrix is ​​multiplied on the right by the third permutation matrix to obtain the first sub-parameter matrix after the initial transformation. Then, the first parameter matrix after the initial transformation is bitwise multiplied with the confusion matrix to obtain the transformed first sub-parameter matrix.

[0260] There may be multiple first sub-parameter matrices in the attention layer, and for different first sub-parameter matrices, the confusion matrices used to transform the first sub-parameter matrices are different.

[0261] The confusion matrix is ​​the interference coefficient in the first transformation function corresponding to the first sub-parameter matrix. By transforming the first sub-parameter matrix using the confusion matrix, the difficulty of cracking the first sub-parameter matrix can be reduced.

[0262] S404, for the second sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the second sub-parameter matrix, the second sub-parameter matrix is ​​transformed using the transpose matrix of the first permutation matrix and the fourth permutation matrix to obtain a transformed second sub-parameter matrix.

[0263] For example, the first sub-parameter matrix is ​​transposed by the first permutation matrix on the left side, and right-multiplied by the fourth permutation matrix to obtain a transformed second sub-parameter matrix.

[0264] S405, for the third sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the third sub-parameter matrix, the third sub-parameter matrix is ​​transformed using the transposed matrix of the first permutation matrix and the fourth permutation matrix to obtain a transformed third sub-parameter matrix.

[0265] Among them, the fourth permutation matrix is ​​equivalent to an added interference coefficient. Using two permutation matrices to transform the second sub-parameter matrix and the third sub-parameter matrix can increase the difficulty of cracking the second sub-parameter matrix and the third sub-parameter matrix.

[0266] For example, the third sub-parameter matrix is ​​multiplied on the left by the transposed matrix of the fourth permutation matrix, and on the right by the first permutation matrix. In the present application, the second sub-parameter matrix and the third sub-parameter matrix may be two sub-parameter matrices that have a multiplication operation in the attention layer, based on which, after the fourth permutation matrix is ​​multiplied by the transposed matrix of the fourth permutation matrix, the interference coefficient added by the fourth permutation matrix can be offset.

[0267] S406: For the first normalized parameter in the attention layer, based on the second transformation function corresponding to the first normalized parameter, the first normalized parameter is transformed using the first permutation matrix to obtain a transformed first normalized parameter.

[0268] For example, the first normalized parameter may be right-multiplied by the first permutation matrix to obtain a transformed first normalized parameter.

[0269] S407, deploying the transformed model parameters in each target layer including the attention layer to the user device.

[0270] It can be understood that the present application selects the first sub-parameter matrix, transforms the first sub-parameter matrix through the confusion matrix, and then combines the functional relationship between the model parameters in the attention layer and the functional relationship between the model parameters in other layers of the target neural network model, which can ultimately offset the influence of the confusion matrix and improve the security of the first sub-parameter matrix.

[0271] Furthermore, considering that the client can obtain the position encoding of each character in the sequence of the input model, as well as the dimensions of the input layer and the attention layer, in order to further reduce the difficulty of the client to crack the model parameters in the attention layer, in the present application, the first position matrix and the second position matrix corresponding to different position indexes in the sequence can also be determined. Among them, the sequence is the sequence that the target neural network model needs to process. The sequence can be composed of multiple word segments, and the position of each word segment in the sequence corresponds to a unique position index. In practical applications, the length of the sequence input to the neural network model can be of various lengths, but in order to be able to determine the position matrix corresponding to each position index, the sequence here can be understood as the maximum length of the sequence supported by the target neural network input, so the sequence can correspond to all possible position indexes.

[0272] On this basis, for each position index, the first position matrix and the second position matrix are transformed using the third permutation matrix and the confusion matrix to obtain the transformed first position matrix and the second position matrix. For example, when the target neural network model is an LLaMA model or other models similar to the LLaMA model, the first position matrix and the second position matrix can be position matrices determined by rotating the position code.

[0273] Accordingly, the transformed first position matrix and the second position matrix corresponding to different position indexes can be deployed to the inference program of the user device, wherein the inference program is used to call the model parameters of the target neural network model for inference calculation.

[0274] Among them, when an inference program is deployed on the user device side, and the inference program includes the transformed first position matrix and second position matrix corresponding to different position indexes, after obtaining the input sequence to be processed, the user device side can query the transformed first position matrix and second position matrix corresponding to the corresponding position index through the inference program based on the position index corresponding to each word segment in the input sequence, without having to calculate the first position matrix and the second position matrix based on the target neural network model. Accordingly, based on the transformed first position matrix and the second position matrix, and by calling the model parameters deployed by the target neural network model to process the input sequence, the interference effect of the confusion matrix can be offset in the end, and the output result corresponding to the input sequence can be obtained.

[0275] like Figure 5 , shows another example diagram of deploying model parameters on the user equipment side in the direction of model provision in an embodiment of the present application.

[0276] Depend on Figure 5It can be seen that the model provider uses two strategies, namely, substitution and obfuscation, to transform the model parameters. On this basis, the model provider will deploy the transformed model parameters to the user device.

[0277] At the same time, the model provider will also pre-calculate the first position matrix and the second position matrix corresponding to different position indexes, and transform the first position matrix and the second position matrix using the permutation and confusion strategies, and then deploy the transformed first position matrix and the second position matrix into the inference program, and deploy the inference program to the user device.

[0278] On this basis, when the user device needs to use the neural network model to process a business request, it can use the inference program to determine the transformed first position matrix and second position matrix corresponding to the position index of each word in the input sequence based on the input sequence included in the business request, and call the model parameters of the deployed target neural network model (the model parameters are transformed model parameters) in combination with the transformed first position matrix and second position matrix to determine the task processing result of the input sequence. Among them, the type of task processing result will be different depending on the business request. For example, if the business processing request is to determine the reply information, the task processing result is the reply content of the input sequence.

[0279] It is understandable that in Figure 4 In the embodiment of, if the target neural network model also includes a feedforward layer, the present implementation can also determine the feedforward layer as the target layer to be protected. Accordingly, for the first feedforward parameter matrix in the feedforward layer, the first feedforward transformation function corresponding to each first feedforward parameter matrix can be determined. The first feedforward transformation function can include at least a first permutation matrix. On this basis, based on the first feedforward transformation function, the first permutation matrix can be used to perform parameter transformation on the first feedforward transformation matrix.

[0280] Furthermore, in order to improve the security of the first feedforward parameter matrix and to ensure that the output result of the target neural network model remains unchanged after the first feedforward parameter matrix is ​​transformed, the first feedforward parameter matrix may include a first sub-feedforward parameter matrix, a second sub-feedforward parameter matrix and a third feedforward parameter matrix.

[0281] Wherein, for the first sub-feedforward parameter matrix, based on the first feedforward transformation function of the first sub-feedforward parameter matrix, the first sub-feedforward parameter matrix is ​​subjected to parameter transformation using the transposed matrix of the first permutation matrix.

[0282] For the second sub-feedforward parameter matrix, based on the first feedforward transformation function of the second sub-feedforward parameter matrix, the second sub-feedforward parameter matrix is ​​parameter-transformed using the transposed matrix of the first permutation matrix and the fifth permutation matrix.

[0283] For the third sub-feedforward parameter matrix, based on the first feedforward transformation function of the third sub-feedforward matrix, the first permutation matrix and the transposed matrix of the fifth permutation matrix are used to perform parameter transformation on the third sub-feedforward parameter matrix.

[0284] Of course, this embodiment can also transform the first feedforward normalization parameter in the feedforward layer. For example, the first feedforward normalization parameter can be transformed based on the first permutation matrix. The details are as described in other embodiments above and will not be repeated here.

[0285] In order to further improve the security of the model parameters in the target neural network model, the present application can also determine all layers in the target neural network model as target layers to be protected. Based on this, the target layer can also include the input layer and output layer of the target neural network model.

[0286] For the first input layer parameter in the input layer, a first input transformation function corresponding to the first input layer parameter may be determined. The first input transformation function may include the first permutation matrix. Accordingly, the first input layer parameter may be transformed using the first permutation matrix.

[0287] Similarly, for the first output layer parameters in the output layer, a first output transformation function corresponding to the first output layer parameters can be determined.

[0288] Furthermore, in order to improve the security of the first output layer parameters, the first output transformation function may include the first permutation matrix and a special permutation matrix that needs to be deployed to the user equipment end. Accordingly, based on the first output transformation function corresponding to the first output layer parameters, the first output layer parameters are transformed based on the first permutation matrix in the first output transformation function and the special permutation matrix to obtain the transformed first output layer parameters.

[0289] On this basis, in the case where the transformed first output layer parameters are deployed to the user equipment side, the present application also needs to deploy the special permutation matrix to the user equipment side.

[0290] Correspondingly, after the inference program of the user device queries the first position matrix and the second position matrix corresponding to the position index of each word segment in the input sequence, it combines the first position matrix and the second position matrix, calls the deployed transformed model parameters, and determines the output result. Then, the user device also needs to use the deployed special permutation matrix to permute the output result to obtain the final inference result.

[0291] For ease of understanding Figure 4 and Figure 5 The specific implementation is described below by taking the first type of neural network model as the LLaMA model as an example.

[0292] It can be understood that the LLaMA model introduces a rotational position encoding to replace the absolute position encoding of the Transormer model. The rotational position encoding is directly applied to the original query vector Q and key value vector K in the attention layer.

[0293] Among them, rotational position coding encodes position information by rotating a vector on a two-dimensional plane. A coding vector f(Q,m) corresponding to each position index is realized by rotating the representation of different dimensions of the input vector at a specific angle, which can be expressed as follows:

[0294]

[0295] Among them, m represents the position index of the word in the sequence. θ is the position encoding cardinality, which is a set value. Q0 and Q1 represent the two-dimensional features of the query vector in the two-dimensional plane. d-1 It represents the features in the d-1 dimension. As mentioned earlier, d is the feature dimension of the attention layer, and the others are similar and will not be repeated here.

[0296] Writing f(q,m) in matrix form can be expressed as:

[0297]

[0298] Among them, the matrix on the left is a position matrix R corresponding to Q q .

[0299] Due to R q With sparseness, R q Convert from matrix form to the following bitwise multiplication operation:

[0300]

[0301] Here, ⊙ represents bitwise matrix multiplication.

[0302] The above is to get a position matrix R q The process is illustrated as an example. Similarly, corresponding to the key value vector K, another position code R corresponding to the position index can be derived K On this basis, the implementation of rotation position encoding in the attention layer can be equivalent to the following:

[0303]

[0304] It is understandable that if a random permutation matrix π1 is added to the above equation, the above equation becomes: (R q π1Q) T (R K π1K)=Q T π1T R q T R K π1K, which will result in π1 T and π1 cannot be eliminated, resulting in that after the attention model parameters are transformed, the output result of the attention layer is inconsistent with the output result before the model transformation, causing the output result of the neural network model to be changed.

[0305] Based on this, the present application considers using the permutation matrix π1 to transform R q , R K By permuting, we can get: R q π1 T π1Q=R q Q, and R K π1 T π1K=R K K.

[0306] Since the permutation matrix π1 cannot be informed to the user device, in order to achieve R q and R K To perform replacement, this application can calculate R in advance for different position indexes q π1 T and R K π1 T , and the transformed R corresponding to different position indexes q and R K Deployed to the inference program on the user device, the R corresponding to different position indexes q π1 T and R K π1 T Deployed to the inference program on the user's device.

[0307] It is understandable that due to R q It is only related to the position index m, the position encoding cardinality θ and the feature dimension d. Therefore, if it is only based on the permutation matrix π1 on R q After permutation, the user device can calculate R q , and R is calculated on the user device side q After that, it is not difficult to deduce R q π1 T Based on this, in order to ensure the privacy of model parameters and improve the security of model parameters, this application can also add a confusion vector g (equivalent to a confusion matrix). d-1 Each element in is multiplied by different confusion factors g0…g in the confusion vector g d-1 , for R q Pre-generated cosmθ j and sinmθj (j is a natural number from 0 to d-1) multiplied by the reciprocal of g 1 / g0…1 / g d-1 , then the influence of the confusion vectors can be eliminated when multiplying by bit, that is, R q The bitwise multiplication operation converted from matrix form can be changed as follows:

[0308]

[0309] From the above formula, we can see that the confusion factors in the confusion vector can cancel each other out during the bitwise multiplication operation. q The replacement of R K It is similar and I will not go into details.

[0310] Based on the above derivation, for the LLaMA model, this application can randomly generate four d×d order permutation matrices: π, π t , π n , π l , and randomly generate a special permutation matrix π of order s×s c In addition, it is necessary to randomly generate a d×d order confusion matrix M q 、M k . As mentioned above, s is the length of the vocabulary.

[0311] On the basis of the above, first, for different position indexes that may exist in the input sequence, for each position index, based on the position index, the position encoding cardinality θ and the feature dimension d, a d×d-order first position matrix R is generated respectively. Q and the second position matrix R k .

[0312] Secondly, this application uses the permutation matrix π t R q , R K To replace, specifically, R q The first position matrix R′ obtained after permutation q , R K After substitution, we get R′ k ,

[0313] Finally, using the confusion matrix M q 、M k R′ q , R′ K Perform obfuscation, that is, calculate R″ q =R′ q ⊙M q , R″ K =R′ K ⊙M k, where R″ q , R″ K The transformed first position matrix and the second position matrix are obtained respectively, and correspondingly, R″ q , R″ K It will be compiled into the inference program and deployed to the user device.

[0314] In addition, the model parameters of each layer in the LLaMA model need to be replaced and confused in the following ways:

[0315] For the input layer parameter W of the s×d order in the input layer of the LLaMA model e , use the following transformation function to transform and obtain the transformed input layer parameter W e ′:

[0316] W e ′=W e π;

[0317] For each parameter matrix W of the attention layer q , W k , W v 、W0:

[0318] First, the matrix parameters of the attention layer are permuted according to the following functions:

[0319] W′ q =π T W q π t ;

[0320] W′ k =π T W k π t ;

[0321] W v ′=π T W v π n ;

[0322]

[0323] Among them, π T , are π and π respectively. n The transpose of .

[0324] Secondly, for W′ q and W′ k To confuse, specifically calculate: W″ q =W′ q ⊙M q , W″ k=W′ k ⊙M k , where W″ q and W k W q , W k The corresponding parameter matrix needs to be deployed to the user device.

[0325] The normalized parameter vector γ1 in the attention layer is permuted to obtain the transformed normalized parameter γ′1: γ′1=γ1π.

[0326] The parameter matrices W1 and W3 in the feedforward layer are permuted to obtain matrix parameters W′1 and W′3, which are as follows:

[0327] W′1=π T W1;

[0328] W′3=π T W3π l ;

[0329] Permutate the parameter matrix W2 in the feedforward layer to obtain the permuted matrix parameter W′2 of W2:

[0330] The normalized parameter vector γ2 in the feedforward layer is permuted to obtain γ′2. Specifically: γ′2=γ2π;

[0331] The output layer parameters W of the output layer c Substitute to get W′ c , specifically: W′ c =π T W c π c .

[0332] On this basis, the model provider will transform the model parameters obtained in each layer, specifically: {W′ e ,W″ q ,W″ k ,W′ v ,W′0,γ′1,W′1,W′2,W′3,γ′2,W′ c} and the special permutation matrix π c Deploy to user devices.

[0333] On the basis of the above, when the user device obtains the business processing request to be processed, it determines the position index corresponding to each word in the input sequence based on the input sequence in the business processing request, and uses the inference program to query the first position matrix and the second position matrix corresponding to the position index of each word.

[0334] Then, the inference program of the user device calls the deployed model parameters and combines the first position sequence and the second position sequence corresponding to each position index to obtain the output result of the target neural network model. c Perform permutation processing on the output result to obtain the final inference result. For example, the inference result is the output result multiplied by π on the right. c The transpose of .

[0335] It is understandable that in Figure 4 In the embodiment, if the target neural network model is a second type neural network model, then Figure 2 In the embodiment, the specific implementation of the transformation function of the target layer to be protected and the model parameters of each layer in the target layer in the second type of neural network model is determined, and each model parameter in the target layer is transformed. The details are as described above and will not be repeated here.

[0336] In another possible implementation, in response to the target neural network model being a second type neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and a third transformation function corresponding to each second parameter matrix in the attention layer and a fourth transformation function corresponding to each second normalization parameter in the attention layer are determined. The third transformation function and the fourth transformation function include a second permutation matrix.

[0337] On this basis, for each second parameter matrix in the attention layer, based on the third transformation function of the second parameter matrix, the second parameter matrix is ​​transformed using the second transposition matrix to obtain a transformed second parameter matrix. Similarly, for the second normalized parameter in the attention layer, based on the fourth transformation function corresponding to the second normalized parameter, the second normalized parameter is transformed using the second permutation matrix to obtain a transformed second normalized parameter.

[0338] In particular, when the target neural network model is a second type neural network model, if the target neural network model includes an input layer, an embedding layer, and an output layer, some or all of the input layer, the embedding layer, and the output layer may be determined as target layers to be protected. The process of transforming the model parameters in any of the input layer, the embedding layer, and the output layer may refer to Figure 4 The process of transforming the relevant model parameters in the input layer, embedding layer and output layer in the embodiment will not be repeated here.

[0339] For ease of understanding, the second type of neural network model is still taken as the Transformer model. The model parameters of each layer in the Transformer can be transformed in the following ways:

[0340] First, randomly generate d×d order permutation matrices π, πt , π n , π l , randomly generate a special permutation matrix π of order s×s c .

[0341] Secondly, for the input layer parameter W in the input layer of the Transformer model e Substitute to get W′ e , W′ e =W e π.

[0342] For the parameter matrix W in the attention layer in the Transformer model q , W k , W v , W0 are replaced to obtain W′ q ,W′ k ,W′ v ,W′0. Specifically:

[0343] W′ q =π T W q π t ;

[0344] W′ k =π T W k π t ;

[0345] W′ v =π T W v π n ;

[0346]

[0347] The normalized parameters λ1, β1 in the attention layer are replaced to obtain γ′1 and β′1, specifically:

[0348] γ′1=γ1π;

[0349] β′1=β1π.

[0350] The parameter matrices W1 and W2 of the feedforward layer in the Transformer model are permuted to obtain W′1 and W′2, respectively, as follows:

[0351] W′1=π T W1π l ;

[0352]

[0353] The normalized parameters λ2 and β2 in the feed-forward layer of the Transformer model are replaced to obtain γ′2 and β′2, respectively, as follows:

[0354] γ′2=γ2π;

[0355] β′2=β2π.

[0356] The output layer parameter W of the output layer in the Transformer model c Substitute to get W′ c , specifically: W′ c =π T W c π c .

[0357] On the basis of the above, the model provider converts the transformed model parameters in each layer of the Transformer model, namely {W′ e ,W′ q ,W′ k ,W′ v ,W′0,γ′1,β′1,W′1,W′2,λ′2,β′2,W′ c} and the special permutation matrix π c Deploy to the user device. Since the dimensions of the permuted model parameters are consistent with the original model parameters, the size of the Transformer model remains unchanged, and the inference process also remains unchanged.

[0358] On the user device side, the transformed model parameters in the Transformer model are used for inference. After the output result is obtained, the output result can be permuted using a special transposed matrix to obtain the final inference result.

[0359] The following introduces the model data processing method from the user device side.

[0360] like Figure 6 , shows a flow chart of another data processing method provided by an embodiment of the present application. This embodiment is applied to a user equipment. The method of this embodiment may include:

[0361] S601, obtaining a model parameter set of a target neural network model sent by a model provider.

[0362] Among them, the target neural network model is the neural network model that needs to be protected.

[0363] The model parameter set includes the transformed model parameters in at least one target layer to be protected in the target neural network model. The transformed model parameters in the target layer are obtained by performing parameter transformation on the model parameters based on the transformation function corresponding to the model parameters.

[0364] Among them, the specific implementation of determining the target neural network model, the target layer to be protected in the target neural network model, and the transformation function of each model parameter in the target layer can be found in the introduction of the previous model provider's related embodiments, which will not be repeated here.

[0365] It can be understood that if the target neural network model has other layers besides the target layer, the model parameter set can also include the model parameters in the other layers besides the target layer in the target neural network model. At this time, the model parameters in the other layers can be untransformed model parameters.

[0366] S602: deploy the model parameter set in the user equipment.

[0367] It can be understood that the inference program sent by the model provider is obtained and deployed in the user device, and the inference program is used to call each model parameter of the target neural network model in the model parameter set.

[0368] In an embodiment of the present application, the model parameters in the target layer that need to be protected in the target neural network model in the model parameter set received on the user device side are all model parameters obtained through transformation. Therefore, it is possible to prevent the user device side from cracking the model parameters, thereby improving the security of the model parameters of the target neural network model.

[0369] Further, in order to more reasonably deploy model parameters for the user equipment side, in the present application, the user equipment may also determine the model requirement information of the user equipment. The model requirement information may include: at least one of the service category information that the user equipment needs to process using the model and the computing capability information of the user equipment. For the model requirement information, please refer to the relevant introduction of the previous embodiment, which will not be repeated here.

[0370] On this basis, the user device can send a model protection request to the model provider based on the model requirement information, and the model protection request includes the model requirement information. Accordingly, the user device can obtain the model parameter set of the target neural network model returned by the model provider in response to the model protection request.

[0371] In the present application, the target layer in the target neural network includes at least the attention layer of the target neural network. The model parameters in the attention layer include a parameter matrix and a normalization matrix. The parameter matrix includes a first sub-parameter matrix, a second sub-parameter matrix and a third sub-parameter matrix.

[0372] In particular, when the transformed first parameter matrix in the attention layer is obtained by transforming the first sub-parameter matrix using the transpose matrix of the first permutation matrix and the third permutation matrix, and performing a secondary transformation on the transformed first sub-parameter matrix in combination with the confusion matrix, the inference program deployed in the user device may include the transformed first position matrix and the second position matrix corresponding to different position indexes.

[0373] Correspondingly, after obtaining a business processing request, the user device can determine the input sequence to be processed in the business processing request, determine the first position matrix and the second position matrix corresponding to the position indexes of different word segments in the input sequence through an inference program, and based on the first position matrix and the second position matrix corresponding to the position index of each word segment, call the model parameter set to determine the output result corresponding to the input sequence.

[0374] The present application also provides an electronic device. Figure 7 As shown, it shows a schematic diagram of a composition structure of the electronic device, and the electronic device at least includes a processor 701 and a memory 703;

[0375] The processor 701 is used to execute the model data processing method described in any one of the above embodiments;

[0376] The memory 702 is used to store the program required by the processor to perform operations

[0377] It is understandable that the electronic device may further include a display unit 703 and an input unit 704 .

[0378] Of course, the electronic device may also have Figure 7 There may be more or fewer components, without limitation.

[0379] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any model data processing method provided in the embodiment of the present application.

[0380] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any model data processing method provided in an embodiment of the present application.

[0381] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0382] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0383] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0384] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A model data processing method, include: Based on the model protection request, determining a target neural network model to be protected, wherein the target neural network model includes multiple layers; Determine at least one target layer to be protected among the multiple layers and a transformation function corresponding to each model parameter in the target layer; Based on the transformation function corresponding to the model parameters in the target layer, the model parameters are transformed to obtain transformed model parameters; The transformed model parameters in each of the target layers are deployed to the user device end.

2. The model data processing method according to claim 1, wherein the determining of at least one target layer to be protected among the multiple layers and the transformation function corresponding to each model parameter in the target layer, include: Based on the model category of the target neural network model, at least one target layer to be protected among the multiple layers and a transformation function corresponding to each model parameter in the target layer are determined.

3. The model data processing method according to claim 2, wherein the at least one target layer to be protected among the multiple layers and the transformation function corresponding to each model parameter in the target layer are determined based on the model category of the target neural network model. include: In response to the target neural network model being a first type neural network model, at least an attention layer in the target neural network model is determined as a target layer to be protected, and a first transformation function corresponding to each first parameter matrix in the attention layer and a second transformation function corresponding to each first normalization parameter in the attention layer are determined, wherein the first transformation function includes a first permutation matrix and a set interference coefficient, and the second transformation function includes the first permutation matrix; In response to the target neural network model being a second type of neural network model, at least the attention layer in the target neural network model is determined as the target layer to be protected, and a third transformation function corresponding to each second parameter matrix in the attention layer and a fourth transformation function corresponding to each second normalization parameter in the attention layer are determined, wherein the third transformation function and the fourth transformation function include a second permutation matrix.

4. The model data processing method according to claim 3, if the target neural network model is a first type neural network model, the first parameter matrix in the attention layer includes a first sub-parameter matrix, a second sub-parameter matrix and a third sub-parameter matrix; The model parameters are transformed based on the transformation function corresponding to the model parameters in the target layer, include: For a first sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the first sub-parameter matrix, the first sub-parameter matrix is ​​transformed using a random number and a transposed matrix of a first permutation matrix; For a second sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the second sub-parameter matrix, transform the second sub-parameter matrix using a random number, a transposed matrix of the first permutation matrix, and a first reversible matrix; For the third sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the third sub-parameter matrix, the third sub-parameter matrix is ​​transformed using random numbers, a first permutation matrix and an inverse matrix corresponding to the first reversible matrix.

5. According to the model data processing method of claim 3, if the target neural network model is a first type neural network model, the first parameter matrix in the attention layer includes a first sub-parameter matrix, a second sub-parameter matrix and a third sub-parameter matrix; The model parameters are transformed based on the transformation function corresponding to the model parameters in the target layer, include: For a first sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the first sub-parameter matrix, the first sub-parameter matrix is ​​transformed using a transposed matrix of the first permutation matrix and a third permutation matrix, and the transformed first sub-parameter matrix is ​​secondarily transformed in combination with a confusion matrix; For a second sub-parameter matrix in the attention layer, based on a first transformation function corresponding to the second sub-parameter matrix, transform the second sub-parameter matrix using a transposed matrix of the first permutation matrix and a fourth permutation matrix; For the third sub-parameter matrix in the attention layer, based on the first transformation function corresponding to the third sub-parameter matrix, the third sub-parameter matrix is ​​transformed using the transposed matrix of the first permutation matrix and the fourth permutation matrix.

6. The model data processing method according to claim 3, if the target neural network model is a second type neural network model, the second parameter matrix in the target neural network model includes a fourth sub-parameter matrix and a fifth sub-parameter matrix; The transformation function corresponding to the model parameters in the target layer is used to transform the model parameters to obtain transformed model parameters. include: For a fourth sub-parameter matrix in the attention layer, based on a third transformation function corresponding to the fourth sub-parameter matrix, the fourth sub-parameter matrix is ​​transformed using a random number, a second permutation matrix, and a second reversible matrix; For the fifth sub-parameter matrix in the attention layer, based on the third transformation function corresponding to the fifth sub-parameter matrix, the fifth sub-parameter matrix is ​​transformed using random numbers, a second permutation matrix and an inverse matrix of the second reversible matrix.

7. The model data processing method according to claim 1, wherein the target neural network model to be protected is determined based on the model protection request. include: Determine model requirement information based on the model protection request, where the model requirement information includes at least one of: service category information that needs to be processed by the model and computing capability information of a user device on which the model is to be deployed; Based on at least one of the business category information and the computing capability information, a target neural network model to be protected is determined.

8. A model data processing method, applied to a user device, include: Obtaining a model parameter set of a target neural network model sent by a model provider, the model parameter set including transformed model parameters in at least one target layer to be protected in the target neural network model, the transformed model parameters in the target layer being obtained by performing parameter transformation on the model parameters based on a transformation function corresponding to the model parameters; The set of model parameters is deployed in the user equipment.

9. The model data processing method according to claim 8, wherein the step of obtaining a model parameter set of a target neural network model sent by a model provider is: include: Determining model requirement information of the user equipment, the model requirement information comprising: at least one of service category information that the user equipment needs to process using the model and computing capability information of the user equipment; Sending a model protection request to a model provider, wherein the model protection request includes the model requirement information; Obtain a model parameter set of the target neural network model returned by the model provider in response to the model protection request.

10. An electronic device comprising a processor and a memory; The processor is used to execute the model data processing method according to any one of claims 1 to 9 above; The memory is used to store programs required for the processor to perform operations.

Citation Information

Cited By

  • Ai model Anti-piracy mechanism

    EP4752761A1

  • Ai model Anti-piracy mechanism

    WO2026115068A1