An information processing method and apparatus

Abstract

Embodiments of the present application disclose an information processing method, comprising: obtaining an initial model, and obtaining hardware configuration information of a plurality of target devices; determining second parameter information corresponding to each target device based on first parameter information of the initial model and the hardware configuration information; and performing hierarchical compression on the initial model in a pipeline manner based on the plurality of second parameter information and the first parameter information to obtain a first model corresponding to each layer, wherein the first model corresponding to a current layer is generated based on model parameters of a previous layer. Embodiments of the present application also disclose an information processing device.

Description

Technical Field

[0001] This application relates to information processing technology in the field of information processing, and more particularly to an information processing method and apparatus. Background Technology

[0002] With the popularization of deep learning, various deep learning models have been widely used in fields such as computer vision and speech recognition. However, when deploying deep learning models, only small models with relatively simple structures and lightweight features can be deployed for some target devices with limited resources. However, since small models have difficulty learning a large amount of knowledge on their own during training with a small amount of labeled sample data, the accuracy of the trained small models is low and it is difficult to meet the deployment requirements. Summary of the Invention

[0003] To address the aforementioned technical problems, this application aims to provide an information processing method and apparatus that solves the problem in related technologies where the accuracy of small models trained is low and difficult to meet deployment requirements.

[0004] The technical solution of this application is implemented as follows:

[0005] An information processing method, the method comprising:

[0006] Obtain the initial model and the hardware configuration information of multiple target devices;

[0007] Based on the first parameter information of the initial model and the hardware configuration information, the second parameter information corresponding to each target device is determined;

[0008] Based on multiple second parameter information and the first parameter information, the initial model is compressed in a pipeline manner to obtain the first model corresponding to each layer; wherein, the first model corresponding to the current layer is generated based on the model parameters corresponding to the previous layer.

[0009] In the above scheme, the step of performing layered compression on the initial model using a pipelined approach based on multiple second parameter information and the first parameter information to obtain a first model corresponding to each layer includes:

[0010] Obtain the clipping and compression relationship corresponding to each layer; wherein, the clipping and compression relationship is the correspondence between the number of compressions and the clipping ratio;

[0011] Based on the number of first parameters of the initial model, the number of second parameters corresponding to each layer, and the pruning and compression relationship, the number of third parameters corresponding to each layer compression is determined; wherein, the first parameter information includes the number of first parameters; and the second parameter information includes the number of second parameters.

[0012] Based on the number of the third parameter, the initial model is compressed in a layered manner using a pipeline method to obtain the first model corresponding to each layer.

[0013] In the above scheme, the step of performing layered compression on the initial model using a pipelined approach based on the number of the third parameter to obtain the first model corresponding to each layer includes:

[0014] Based on the number of third parameters corresponding to the first layer of compression, the initial model is periodically compressed to obtain the first model corresponding to the first layer;

[0015] In the process of periodically compressing the initial model to obtain the first model corresponding to the first layer, based on the number of third parameters corresponding to the second layer compression, the first target compressed sub-model corresponding to the first layer is periodically compressed to obtain the first model corresponding to the second layer, until the process of obtaining the first model corresponding to the (N-1)th layer is periodically compressed to obtain the first model corresponding to the Nth layer; wherein, the first target compressed sub-model is obtained by compressing the initial model; or,

[0016] Based on the number of third parameters corresponding to the second layer compression, the first model corresponding to the first layer is periodically compressed to obtain the first model corresponding to the second layer, until the first model corresponding to the (N-1)th layer is compressed to obtain the first model corresponding to the Nth layer; where N is a positive integer greater than 1.

[0017] In the above scheme, the step of periodically compressing the initial model based on the number of third parameters corresponding to the first layer compression to obtain the first model corresponding to the first layer includes:

[0018] When the model parameters of the initial model are updated for the Mth time to obtain the Mth updated model, the Mth updated model is compressed for the first time based on the number of third parameters corresponding to the first compression of the first layer to obtain the first first compressed sub-model corresponding to the first layer. At the same time, the model parameters of the Mth updated model are updated for the M+1th time to obtain the M+1th updated model.

[0019] Based on the number of third parameters corresponding to the second compression of the second layer and the (M+1)th update model, the first first compression sub-model is compressed a second time to obtain the second first compression sub-model corresponding to the first layer. At the same time, the (M+1)th update model is updated for the (M+2)th time to obtain the (M+2)th update model. This process continues until the number of parameters of the i-th first compression sub-model corresponding to the first layer satisfies the number of second parameters corresponding to the first layer. The i-th first compression sub-model is then used as the first model. Here, M and i are positive integers.

[0020] In the above scheme, the step of periodically compressing the initial model based on the number of third parameters corresponding to the first layer compression to obtain the first model corresponding to the first layer includes:

[0021] Based on the number of third parameters corresponding to the first compression of the first layer, the initial model is compressed for the first time to obtain the first compressed sub-model corresponding to the first layer.

[0022] Based on the number of third parameters corresponding to the second compression of the first layer and the initial model, the first first compression sub-model is compressed a second time to obtain the second first compression sub-model corresponding to the first layer, until the number of parameters of the i-th first compression sub-model satisfies the number of second parameters corresponding to the first layer, and the i-th first compression sub-model is taken as the first model corresponding to the first layer; where i is a positive integer.

[0023] In the above scheme, the step of periodically compressing the initial model to obtain the first model corresponding to the first layer, and periodically compressing the first target compressed sub-model corresponding to the first layer based on the number of third parameters corresponding to the second layer compression to obtain the first model corresponding to the second layer, includes:

[0024] When the initial model is compressed for the jth time to obtain the jth first compressed sub-model corresponding to the first layer, the jth first compressed sub-model is compressed for the first time based on the number of third parameters corresponding to the first compression of the second layer to obtain the 1st second compressed sub-model corresponding to the second layer. At the same time, the initial model is compressed for the (j+1)th time to obtain the (j+1)th first compressed sub-model.

[0025] Based on the number of third parameters corresponding to the second compression of the second layer and the (j+1)th first compression sub-model, the first second compression sub-model is compressed a second time to obtain the second second compression sub-model corresponding to the second layer. At the same time, the initial model is compressed j+2 times to obtain the (j+2)th first compression sub-model. This process continues until the number of parameters of the obtained h-th second compression sub-model satisfies the number of second parameters corresponding to the second layer. The h-th second compression sub-model is then used as the first model corresponding to the second layer. Here, h and j are both positive integers.

[0026] In the above scheme, the step of performing a first compression on the initial model based on the number of third parameters corresponding to the first compression of the first layer to obtain the first compressed sub-model corresponding to the first layer includes:

[0027] Based on the number of third parameters corresponding to the first compression of the first layer, the initial model is compressed for the first time to obtain the initial compressed sub-model;

[0028] The model parameters of the initial compression sub-model are updated to obtain the first compression sub-model.

[0029] The method in the above scheme further includes:

[0030] Obtain first sample data and process the first sample data using the initial model to obtain first label data;

[0031] Based on the first sample data and the first label data, the first model corresponding to each layer is trained to obtain the second model corresponding to each layer.

[0032] The second model corresponding to each layer is trained a second time to obtain the target model corresponding to each layer.

[0033] In the above scheme, the step of performing a second training on the second model corresponding to each layer to obtain the target model corresponding to each layer includes:

[0034] Obtain the second sample data and determine the second model corresponding to the first layer as the target model corresponding to the first layer;

[0035] The second sample data is processed using the target model corresponding to the first layer to obtain the second label data. Based on the second label data and the second sample data, the second model corresponding to the second layer is trained again to obtain the target model corresponding to the second layer.

[0036] The second sample data is processed using the target model corresponding to the second layer to obtain the third label data. Based on the third label data and the second sample data, the second model corresponding to the third layer is trained a second time to obtain the target model corresponding to the third layer. This process continues until the second model corresponding to the Nth layer is trained a second time based on the Nth label data and the second sample data to obtain the target model corresponding to the Nth layer.

[0037] An information processing apparatus, the apparatus comprising:

[0038] The acquisition unit is used to acquire the initial model and the hardware configuration information of multiple target devices;

[0039] The processing unit is used to determine the second parameter information corresponding to each target device based on the first parameter information of the initial model and the hardware configuration information;

[0040] The processing unit is further configured to perform layered compression on the initial model in a pipeline manner based on multiple second parameter information and the first parameter information to obtain a first model corresponding to each layer; wherein, the first model corresponding to the current layer is generated based on the model parameters corresponding to the previous layer.

[0041] The information processing method and apparatus provided in the embodiments of this application obtain an initial model and hardware configuration information of multiple target devices; based on the first parameter information and hardware configuration information of the initial model, determine the second parameter information corresponding to each target device; based on multiple second parameter information and first parameter information, perform layered compression of the initial model in a pipelined manner to obtain a first model corresponding to each layer; wherein, the first model corresponding to the current layer is generated based on the model parameters corresponding to the previous layer; thus, by performing layered compression of the initial model in a pipelined manner, the first model corresponding to each layer retains the knowledge of the initial model, and the compression span is small when performing layered compression in a pipelined manner, making it easier to transfer the knowledge of the initial model to the first model corresponding to each layer, unlike related technologies that rely on a small amount of labeled sample data for model training to obtain the first model, thereby improving the accuracy of the obtained first model and solving the problem that the small model trained in related technologies has low accuracy and is difficult to meet deployment requirements. Attached Figure Description

[0042] Figure 1 A flowchart illustrating an information processing method provided in an embodiment of this application;

[0043] Figure 2 A flowchart illustrating another information processing method provided in an embodiment of this application;

[0044] Figure 3 A flowchart illustrating another information processing method provided in an embodiment of this application;

[0045] Figure 4 This is a flowchart illustrating an information processing method according to another embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the structure of an information processing device provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0049] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0050] This application provides an information processing method that can be applied to electronic devices. (Refer to...) Figure 1 As shown, the method includes the following steps:

[0051] Step 101: Obtain the initial model and the hardware configuration information of multiple target devices.

[0052] In this embodiment, the initial model can be a complex and highly accurate model obtained by pre-training a large amount of sample data using a deep learning algorithm. The initial model can be trained by an electronic device; alternatively, it can be trained by other devices and then sent to the electronic device.

[0053] In one feasible implementation, the initial model can be obtained by an electronic device using a self-supervised learning algorithm to train the model on a small amount of labeled sample data and a large amount of unlabeled sample data; wherein, the electronic device can be a server.

[0054] In this embodiment, the hardware configuration of the target device is lower than that of the electronic device; it can also be understood as the computing power of the target device being lower than that of the electronic device. The hardware configuration information of the target device includes the configuration information of the target device's processor, memory, and hard disk. The electronic device can send a request to each target device to obtain hardware configuration information, and the target device can send its own hardware configuration information to the electronic device based on the received request. Of course, the hardware configuration information of each target device can also be obtained based on user input.

[0055] In one feasible implementation, the target device can be an edge device with limited computing resources; the hardware configuration information of the edge device can include the number of CPU cores, memory capacity, and hard disk configuration; and the number of target devices can be three.

[0056] Step 102: Based on the first parameter information and hardware configuration information of the initial model, determine the second parameter information corresponding to each target device.

[0057] In this embodiment, the computing power of each target device can be calculated based on its hardware configuration information; wherein, the computing power of the target device represents its data processing capability; the electronic device can determine the computing power required to run the initial model based on the first parameter information of the initial model; and based on the computing power of each target device, the computing power required to run the initial model, and the first parameter information of the initial model, the second parameter information of the model that each target device can run is determined. The first parameter information may be the number of model parameters in the initial model.

[0058] Specifically, the ratio between the computing power required to run the initial model and the first parameter information of the initial model can be calculated, and based on this ratio and the computing power of each target device, the second parameter information corresponding to each target device can be determined.

[0059] In one feasible implementation, the target device's computing power is A, the initial model has 100 parameters, and the computing power required to run the initial model is B. The second parameter information of the target device can be (100*A) / B = 80. That is, the number of model parameters in the initial model needs to be reduced to 80 before the reduced initial model can be run on the target device. Reducing the number of model parameters to 80 can be understood as selecting the 80 most important model parameters from the initial 100 parameters and labeling the other 20 model parameters as invalid. The invalidated weight parameters do not participate in the forward computation of the data, but they need to be updated during the backward gradient update; updating the weight parameters refers to updating their values.

[0060] Step 103: Based on multiple second parameter information and first parameter information, the initial model is compressed in layers using a pipeline method to obtain the first model corresponding to each layer.

[0061] The first model corresponding to the current layer is generated based on the parameters of the model corresponding to the previous layer. The computing power required to run the first model corresponding to the previous layer is greater than the computing power required to run the first model corresponding to the current layer. The pipelined approach is reflected in the fact that compression is enabled for each layer sequentially, and the compression of the current layer depends on the compression of the previous layer.

[0062] In this embodiment, the model parameters corresponding to the previous layer can be the model parameters of the first model corresponding to the previous layer; alternatively, they can be the model parameters of the compressed sub-model obtained in a certain compression step during the process of determining the first model corresponding to the previous layer through multiple compressions. The electronic device can use a pipelined approach to perform parallel layered compression of the initial model to obtain the first model corresponding to each layer; alternatively, it can use a pipelined approach to perform serial layered compression of the initial model to obtain the first model corresponding to each layer.

[0063] Serial layered compression refers to starting compression of the current layer based on the first model obtained from multiple compressions of the previous layer. Parallel layered compression refers to starting compression of the current layer while the first model obtained from multiple compressions of the previous layer is being obtained (i.e., before the first model is obtained); in other words, compression of the current layer is started based on the compressed sub-model obtained from a single compression operation of the previous layer.

[0064] During parallel layered compression, compression of the current layer begins before compression of the previous layer has finished. During a certain period of time, while the previous layer is compressing, the current layer is also compressing.

[0065] It should be noted that a pipelined approach is used to compress the initial model in layers, resulting in the first model corresponding to each layer. In this way, the pipelined layered compression of the initial model, which is complex and highly accurate, results in a smaller compression span during compression, making it easier to transfer knowledge. Moreover, since the entire process gradually initiates the compression of each layer in a pipelined manner, and the compression of the current layer depends on the compression of the previous layer, the knowledge transfer during the compression process is more solid, and the robustness and generalization are stronger.

[0066] The information processing method provided in the embodiments of this application obtains an initial model and hardware configuration information of multiple target devices; based on the first parameter information of the initial model and the hardware configuration information, it determines the second parameter information corresponding to each target device; based on multiple second parameter information and the first parameter information, it performs layered compression of the initial model in a pipelined manner to obtain the first model corresponding to each layer; wherein, the first model corresponding to the current layer is generated based on the model parameters corresponding to the previous layer; thus, by performing layered compression of the initial model in a pipelined manner, the first model corresponding to each layer retains the knowledge of the initial model, and the compression span is small when performing layered compression in a pipelined manner, making it easier to transfer the knowledge of the initial model to the first model corresponding to each layer. Unlike related technologies, it no longer relies on a small amount of labeled sample data for model training to obtain the first model, thereby improving the accuracy of the obtained first model and solving the problem that the small model trained in related technologies has low accuracy and is difficult to meet deployment requirements.

[0067] Based on the foregoing embodiments, embodiments of this application provide an information processing method that can be applied to electronic devices, see below. Figure 2 As shown, the method includes the following steps:

[0068] Step 201: Obtain the initial model and the hardware configuration information of multiple target devices.

[0069] Step 202: Based on the first parameter information and hardware configuration information of the initial model, determine the second parameter information corresponding to each target device.

[0070] Step 203: Obtain the clipping and compression relationship for each layer.

[0071] The clipping compression relationship is the correspondence between the number of compressions and the clipping ratio; the clipping compression relationship for each layer can be the same or different. The number of target devices is the same as the number of layers involved in the layered compression.

[0072] In this embodiment of the application, the clipping and compression relationship corresponding to each layer can be obtained based on the user's input operation; of course, the clipping and compression relationship corresponding to each layer can also be determined by the electronic device based on the difference between the first parameter information and the second parameter information of the target device corresponding to that layer.

[0073] In one feasible implementation, the clipping and compression relationship for each layer can be the same; the target device and the number of layers are both 3. Specifically, the clipping and compression relationship for each layer can be 5%->10%->15%->…95%->100%; that is, the clipping ratio for the first compression of each layer is 5%, the clipping ratio for the second compression is 10%, and the clipping ratio for the third compression is 15%.

[0074] Step 204: Based on the number of first parameters of the initial model, the number of second parameters corresponding to each layer, and the pruning and compression relationship, determine the number of third parameters corresponding to each layer compression.

[0075] The first parameter information includes the number of first parameters; the second parameter information includes the number of second parameters.

[0076] In this embodiment of the application, the number of first parameters may be the number of model parameters of the initial model; the number of second parameters may be the number of model parameters of the model that the target device can run at each layer.

[0077] In one feasible implementation, the number of first parameters can specifically be the number of weight parameters of the initial model; the number of second parameters can specifically be the number of weight parameters of the model that the target device can run for each layer.

[0078] In this embodiment, the number of third parameters corresponding to the current layer compression is related to the number of third parameters corresponding to the previous layer compression. Each layer needs to be compressed multiple times, and the number of third parameters corresponding to each layer compression can specifically be the number of third parameters corresponding to each compression step of each layer.

[0079] Specifically, the number of third parameters corresponding to the first layer compression can be obtained based on the number of first parameters of the initial model, the number of second parameters corresponding to the first layer, and the pruning and compression relationship corresponding to the first layer; the number of third parameters corresponding to the second layer compression can be obtained based on at least one third parameter corresponding to the first layer compression, the number of second parameters corresponding to the second layer, and the pruning and compression relationship corresponding to the second layer, until the number of third parameters corresponding to the Nth layer compression is obtained based on at least one third parameter corresponding to the N-1th layer, the number of second parameters corresponding to the Nth layer, and the pruning and compression relationship corresponding to the Nth layer.

[0080] For serial layered compression, the third parameter information for the current layer can be determined based on the number of third parameters corresponding to the last compression of the previous layer, the number of second parameters corresponding to the current layer, and the pruning ratio corresponding to the current layer. For parallel layered compression, the third parameter information for the current layer can be determined based on the number of third parameters corresponding to multiple compressions of the previous layer, the number of second parameters corresponding to the current layer, and the pruning ratio corresponding to the current layer.

[0081] In one feasible implementation, the number of first parameters of the initial model is denoted by N0', where N0' = 100; the number of second parameters corresponding to the first layer is 80. The number of third parameters corresponding to each compression of the first layer is denoted by N1', where N1' = N0' - (N0' - 80) * P1; and P1 is the pruning ratio of the first layer. The pruning ratio for the first compression of the first layer is 5%, and the number of third parameters corresponding to the first compression of the first layer is N1' = 100 - (100 - 80) * 5% = 99; the pruning ratio for the second compression of the first layer is 10%, and the number of third parameters corresponding to the second compression of the first layer is N1' = 100 - (100 - 80) * 10% = 98. Regardless of whether serial or parallel layered compression is performed, N0' remains constant.

[0082] The number of second parameters corresponding to the second layer is 60; the number of third parameters corresponding to each compression of the second layer is represented by N2', N2' = N1' - (N1' - 60) * P2; P2 is the clipping ratio of the second layer; for parallel layered compression, N1' can be the number of third parameters corresponding to multiple compressions of the first layer, and for serial layered compression, N1' can be the number of third parameters corresponding to the last compression of the first layer.

[0083] It should be noted that when calculating N1' for each compression in the first layer, the value obtained by (N0'-80)*P1 can be rounded up, and then N0' minus the rounded-up value is used to obtain N1'. Similarly, when calculating N2' for each compression in the second layer, the value obtained by (N1'-60)*P2 can be rounded up, and then N1' minus the rounded-up value is used to obtain N2' for each compression. This avoids the calculated number of third parameters for each layer being a decimal, making each compression in each layer easier to implement.

[0084] It should be noted that the number of third parameters corresponding to each compression layer can be determined based on the number of first parameters of the initial model, the number of second parameters corresponding to each layer, and the compression relationship of each layer before the initial model is compressed in a pipeline manner; of course, it can also be calculated when the current layer is compressed, that is, it is calculated in real time.

[0085] Step 205: Based on the number of the third parameter, the initial model is compressed in layers using a pipeline method to obtain the first model corresponding to each layer.

[0086] In the embodiments of this application, the initial model can be serially or in parallel to be compressed in a pipeline manner to obtain the first model corresponding to each layer; of course, during the training of the initial model, based on the number of the third parameter, the initial model after a certain training can be serially or in parallel to be compressed in a pipeline manner to obtain the first model corresponding to each layer.

[0087] It should be noted that during the training of the initial model, the weight parameters of the initial model are continuously updated, but the number of weight parameters remains unchanged.

[0088] It should be noted that step 205 can be achieved through steps a1~a2 or a1~a3:

[0089] Step a1: Based on the number of third parameters corresponding to the first layer compression, periodically compress the initial model to obtain the first model corresponding to the first layer.

[0090] In the embodiments of this application, when performing the first layer compression (i.e., periodically compressing the initial model), the initial model can be compressed for the first time based on the number of third parameters corresponding to the first compression of the first layer to obtain the first first compressed sub-model. Based on the number of third parameters corresponding to the second compression of the first layer, the first first compressed sub-model is compressed for the second time to obtain the second first compressed sub-model. This process continues until the number of parameters of the obtained first compressed sub-model satisfies the number of second parameters corresponding to the first layer, and then the first compressed sub-model is used as the first model.

[0091] Step a2: In the process of periodically compressing the initial model to obtain the first model corresponding to the first layer, based on the number of third parameters corresponding to the second layer compression, periodically compress the first target compression sub-model corresponding to the first layer to obtain the first model corresponding to the second layer, until in the process of obtaining the first model corresponding to the N-1th layer, periodically compress the N-1th target compression sub-model corresponding to the N-1th layer to obtain the first model corresponding to the Nth layer.

[0092] The first target compression sub-model is obtained by compressing the initial model; specifically, the first target compression sub-model can be the model obtained after a certain compression of the first layer has ended, before the first layer compression has ended.

[0093] In this embodiment, during the process of periodically compressing the initial model to obtain the first model corresponding to the first layer, upon obtaining the first target compressed sub-model corresponding to the first layer, compression of the second layer is initiated based on the first target compressed sub-model. Specifically, this can be done by first compressing the first target compressed sub-model based on the number of third parameters corresponding to the first compression of the second layer, obtaining the first second compressed sub-model; then, based on the number of third parameters corresponding to the second compression of the second layer, the first second compressed sub-model is compressed a second time, obtaining the second second compressed sub-model, and so on, until the number of parameters of the obtained second compressed sub-model satisfies the number of second parameters corresponding to the second layer. This second compressed sub-model is then used as the first model corresponding to the second layer. This process continues until, during the process of obtaining the first model corresponding to the (N-1)th layer, the (N-1)th target compressed sub-model corresponding to the (N-1)th layer is periodically compressed to obtain the first model corresponding to the Nth layer. The (N-1)th target compressed sub-model can be the model obtained after a certain compression of the (N-1)th layer has ended, before the compression of the (N-1)th layer has finished.

[0094] Step a3: Based on the number of third parameters corresponding to the second layer compression, periodically compress the first model corresponding to the first layer to obtain the first model corresponding to the second layer, until the first model corresponding to the (N-1)th layer is compressed to obtain the first model corresponding to the Nth layer.

[0095] Where N is a positive integer greater than 1.

[0096] In this embodiment, the first model corresponding to the first layer can be compressed for the first time based on the number of third parameters corresponding to the first compression of the second layer to obtain the first second compressed sub-model. Based on the number of third parameters corresponding to the second compression of the second layer, the first second compressed sub-model can be compressed for the second time to obtain the second second compressed sub-model. Based on the number of third parameters corresponding to the third compression of the second layer, the second second compressed sub-model can be compressed for the third time to obtain the third second compressed sub-model. This process continues until the number of parameters of the obtained second compressed sub-model meets the number of second parameters corresponding to the second layer. This second compressed sub-model is then used as the first model corresponding to the second layer. This process is repeated until the first model corresponding to the (N-1)th layer is periodically compressed to obtain the first model corresponding to the Nth layer.

[0097] It should be noted that, during the periodic training of the first model corresponding to the first layer, the first model after training at the end of a certain training cycle can be periodically compressed based on the number of the third parameters corresponding to the first compression of the second layer to obtain the first model corresponding to the second layer; until the first model corresponding to the N-1th layer after training is periodically compressed to obtain the first model corresponding to the Nth layer.

[0098] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0099] The information processing method provided in the embodiments of this application uses a pipelined approach to perform layered compression of the initial model. The first model corresponding to each layer retains the knowledge of the initial model. Furthermore, when using a pipelined approach for layered compression, the compression span is small, making it easier to transfer the knowledge of the initial model to the first model corresponding to each layer. Unlike related technologies, it no longer relies on a small amount of labeled sample data for model training to obtain the first model, thus improving the accuracy of the obtained first model. This solves the problem in related technologies where the small models trained have low accuracy and are difficult to meet deployment requirements.

[0100] Based on the foregoing embodiments, embodiments of this application provide an information processing method that can be applied to electronic devices, see below. Figure 3 As shown, the method includes the following steps:

[0101] Step 301: Obtain the initial model and the hardware configuration information of multiple target devices.

[0102] Step 302: Based on the first parameter information and hardware configuration information of the initial model, determine the second parameter information corresponding to each target device.

[0103] Step 303: Obtain the clipping and compression relationship for each layer.

[0104] The cropping compression relationship is the correspondence between the number of compressions and the cropping ratio.

[0105] Step 304: Based on the number of first parameters of the initial model, the number of second parameters corresponding to each layer, and the pruning and compression relationship, determine the number of third parameters corresponding to each layer compression.

[0106] The first parameter information includes the number of first parameters; the second parameter information includes the number of second parameters.

[0107] It should be noted that steps 305-306 or steps 307-308 can be executed after step 304;

[0108] Step 305: After updating the model parameters of the initial model for the Mth time to obtain the Mth updated model, based on the number of third parameters corresponding to the first compression of the first layer, perform the first compression of the Mth updated model to obtain the first first compressed sub-model corresponding to the first layer. At the same time, update the model parameters of the Mth updated model for the M+1th time to obtain the M+1th updated model.

[0109] In this embodiment, during the periodic training of the initial model, when the initial model has been trained M times to obtain the Mth updated model, the compression of the first layer can begin. That is, when the initial model is trained for the (M+1)th time, the first compression of the first layer is also being performed simultaneously. Thereafter, in each training cycle, while training the initial model, the compression of the first layer also needs to be performed until the first model corresponding to the first layer is obtained, at which point the compression of the first layer stops.

[0110] It should be noted that during the periodic training of the initial model, the training sample data used in each training cycle is different; that is, each training cycle has different training sample data. The training sample data includes labeled sample data and unlabeled sample data.

[0111] Step 306: Based on the number of third parameters corresponding to the second compression of the second layer and the M+1th update model, perform a second compression on the first first compression sub-model to obtain the second first compression sub-model corresponding to the first layer. At the same time, perform the M+2th update on the M+1th update model to obtain the M+2th update model. Continue until the number of parameters of the i-th first compression sub-model corresponding to the first layer satisfies the number of second parameters corresponding to the first layer. Then, take the i-th first compression sub-model as the first model.

[0112] Where M and i are positive integers.

[0113] In this embodiment, during the (M+2)th training cycle, when the initial model is trained M+2 times, the second compression of the first layer begins. Specifically, based on the number of third parameters corresponding to the second compression of the first layer and the (M+1)th updated model, the first first compressed sub-model is compressed a second time to obtain the second first compressed sub-model. That is, when the first layer is compressed for the Wth time, the (W-1)th first compressed sub-model needs to be compressed for the Wth time based on the number of third parameters corresponding to the Wth compression of the first layer and the (M+W-1)th updated model obtained from the initial model after the M+W-1th training, to obtain the Wth first compressed sub-model corresponding to the first layer. This continues until the number of parameters of the i-th first compressed sub-model corresponding to the first layer satisfies the number of second parameters corresponding to the first layer, and the i-th first compressed sub-model is used as the first model corresponding to the first layer. During the compression of the (M+W-1)th updated model obtained from the initial model after the M+W-1th training, the initial model needs to be trained for the M+Wth time.

[0114] In one feasible implementation, the number of parameters of the i-th first compression sub-model corresponding to the first layer satisfies that the number of second parameters corresponding to the first layer can be understood as the number of weight parameters of the i-th first compression sub-model being the same as the number of weight parameters corresponding to the target device in the first layer.

[0115] The following section provides a detailed explanation of the implementation process of performing the Wth compression on the Wth first compression sub-model, based on the number of third parameters corresponding to the Wth compression of the first layer and the M+W-1th updated model obtained by training the initial model for the M+W-1th time.

[0116] In this embodiment, the model parameters of the (W-1)th first compression sub-model can be filtered based on the number of third parameters corresponding to the Wth compression in the first layer to obtain the weight parameters corresponding to the Nth compression in the first layer. Then, the weight parameters of the (W-1)th first compression sub-model, excluding the weight parameters corresponding to the Wth compression in the first layer, are labeled as invalid, resulting in the Wth first labeled sub-model. Subsequently, based on the (M+W-1)th updated model and the Wth first labeled sub-model, the Wth first compression sub-model corresponding to the first layer is generated. During the periodic training of the initial model, the weight parameters of the initial model change, but the number of weight parameters remains constant.

[0117] Specifically, based on the M+W-1th update model and the Wth first-labeled sub-model, the Wth first-compression sub-model corresponding to the first layer is generated. This can be achieved by acquiring unlabeled sample data and inputting it into the M+N-1th update model to obtain its output. Then, the unlabeled and labeled sample data are input into the Nth first-labeled sub-model to obtain its output. A loss function is used to calculate the loss value based on the outputs of the M+N-1th update model, the Nth first-labeled model, and the labels corresponding to the labeled sample data. The model parameters of the Nth first-labeled sub-model are then updated based on the obtained loss value to obtain the Nth first-compression sub-model. In the Nth first-labeled sub-model, weight parameters with invalid labels do not participate in the forward computation; however, when updating the model parameters of the Nth first-labeled model, the weight parameters with invalid labels need to be updated.

[0118] It should be noted that steps 309-310 or step 311 can be executed after step 306.

[0119] Step 307: Based on the number of third parameters corresponding to the first compression of the first layer, perform the first compression on the initial model to obtain the first compressed sub-model corresponding to the first layer.

[0120] In this embodiment of the application, the model parameters of the initial model can be filtered based on the number of third parameters corresponding to the first compression of the first layer, so as to obtain the weight parameters with higher importance among the model parameters of the initial model, and the weight parameters of the initial model other than the weight parameters with higher importance are marked as invalid, so as to obtain the first first compression sub-model corresponding to the first layer.

[0121] It should be noted that step 307 can be achieved through steps b1 to b2:

[0122] Step b1: Based on the number of third parameters corresponding to the first compression of the first layer, perform the first compression on the initial model to obtain the initial compressed sub-model.

[0123] In this embodiment, based on the number of third parameters corresponding to the first compression of the first layer, the model parameters of the initial model are filtered to obtain the weight parameters with higher importance among the model parameters of the initial model. The weight parameters of the initial model other than these high-importance weight parameters are then marked as invalid, resulting in an initial compressed sub-model. This initial compressed sub-model can also be referred to as the first labeled sub-model corresponding to the first compression of the first layer. The number of high-importance weight parameters obtained is the same as the number of third parameters corresponding to the first compression of the first layer.

[0124] Step b2: Update the model parameters of the initial compression sub-model to obtain the first compression sub-model.

[0125] In this embodiment, unlabeled sample data can be input into the initial model to obtain its output. Then, unlabeled and labeled sample data can be input into the initial compressed sub-model to obtain its output. A loss function is then used to calculate the loss value based on the output of the initial model, the output of the initial compressed sub-model, and the labels corresponding to the labeled sample data. The model parameters of the initial compressed sub-model are updated according to this loss value to obtain the first compressed sub-model. Notably, weight parameters with invalid labels in the initial compressed sub-model do not participate in forward data calculation; that is, they do not process the unlabeled sample data. However, they participate in backward gradient updates (i.e., when updating the model parameters of the initial compressed sub-model, the weight parameters with invalid labels also need to be updated).

[0126] Step 308: Based on the number of third parameters corresponding to the second compression of the first layer and the initial model, perform a second compression on the first first compression sub-model to obtain the second first compression sub-model corresponding to the first layer, until the number of parameters of the i-th first compression sub-model satisfies the number of second parameters corresponding to the first layer, and take the i-th first compression sub-model as the first model corresponding to the first layer; where i is a positive integer.

[0127] In this embodiment of the application, based on the number of third parameters corresponding to the second compression of the first layer, the first compression sub-model is compressed a second time to obtain the second labeled sub-model of the first layer. Then, based on the initial model, unlabeled sample data and labeled sample data, the model parameters of the second labeled sub-model are updated to obtain the second first compression sub-model corresponding to the first layer.

[0128] Specifically, unlabeled sample data can be input into the initial model to obtain the output of the initial model. Unlabeled sample data and labeled sample data can be input into the second labeled sub-model to obtain the output of the second labeled sub-model. The loss value is calculated based on the output of the initial model, the output of the second labeled sub-model, and the labels corresponding to the labeled sample data using a loss function. The model parameters of the second labeled sub-model are updated based on the loss value to obtain the second first compression sub-model corresponding to the first layer.

[0129] It should be noted that steps 309-310 or step 311 can be executed after step 308.

[0130] Step 309: After compressing the initial model for the jth time to obtain the jth first compressed sub-model corresponding to the first layer, based on the number of third parameters corresponding to the first compression of the second layer, compress the jth first compressed sub-model for the first time to obtain the 1st second compressed sub-model corresponding to the second layer. At the same time, compress the initial model for the (j+1)th time to obtain the (j+1)th first compressed sub-model.

[0131] In this embodiment of the application, based on the number of third parameters corresponding to the first compression of the second layer, the j-th first compression sub-model is compressed for the first time to obtain the marked sub-model corresponding to the first compression of the second layer, and the model parameters of the marked sub-model are updated to obtain the first second compression sub-model corresponding to the second layer.

[0132] Step 310: Based on the number of third parameters corresponding to the second compression of the second layer and the (j+1)th first compression sub-model, perform a second compression on the first second compression sub-model to obtain the second second compression sub-model corresponding to the second layer. At the same time, perform (j+2) compressions on the initial model to obtain the (j+2)th first compression sub-model. Continue until the number of parameters of the obtained h-th second compression sub-model satisfies the number of second parameters corresponding to the second layer. Then, use the h-th second compression sub-model as the first model corresponding to the second layer.

[0133] Where h and j are both positive integers.

[0134] In this embodiment, the first second compression sub-model can be compressed a second time based on the number of third parameters corresponding to the second compression of the second layer to obtain the marker sub-model corresponding to the second compression of the second layer. Then, based on the (j+1)th first compression sub-model, the model parameters of the marker sub-model corresponding to the second compression of the second layer are updated to obtain the second second compression sub-model corresponding to the second layer.

[0135] Step 311: Based on the number of third parameters corresponding to the second layer compression, periodically compress the first model corresponding to the first layer to obtain the first model corresponding to the second layer, until the first model corresponding to the (N-1)th layer is compressed to obtain the first model corresponding to the Nth layer.

[0136] Where N is a positive integer greater than 1.

[0137] Based on the foregoing embodiments, in other embodiments of this application, the method may further include the following steps:

[0138] Step 312: Obtain the first sample data and process the first sample data using the initial model to obtain the first label data.

[0139] The first sample data consists of unlabeled sample data.

[0140] In this embodiment of the application, the first sample data can be input into the initial model, so that the initial model processes the first sample data and outputs the processing result as the first label data.

[0141] Step 313: Based on the first sample data and the first label data, perform the first training on the first model corresponding to each layer to obtain the second model corresponding to each layer.

[0142] In this embodiment, the first sample data can be input into the first model corresponding to each layer to obtain the output result of the first model. A loss function is used to calculate the loss value based on the output result of the first model and the first label data. The model parameters of the first model corresponding to the layer are updated based on the loss value to obtain the second model corresponding to each layer.

[0143] Step 313: Perform a second training on the second model corresponding to each layer to obtain the target model corresponding to each layer.

[0144] The second training can be a distillation training.

[0145] In the embodiments of this application, a target model corresponding to the first layer can be determined. Based on the target model corresponding to the first layer, a second training is performed on the second model corresponding to the second layer to obtain the target model corresponding to the second layer. Based on the target model corresponding to the second layer, a second training is performed on the second model corresponding to the third layer to obtain the target model corresponding to the third layer. This process continues until the target model corresponding to the Nth layer is guided to train based on the target model corresponding to the (N-1)th layer to obtain the target model corresponding to the Nth layer.

[0146] It should be noted that step 313 can be achieved through steps c1 to c3:

[0147] Step c1: Obtain the second sample data and determine the second model corresponding to the first layer as the target model corresponding to the first layer.

[0148] The second sample data can be unlabeled data pre-acquired by the electronic device. The first and second sample data can be the same or different.

[0149] Step c2: Process the second sample data using the target model corresponding to the first layer to obtain the second label data, and perform a second training on the second model corresponding to the second layer based on the second label data and the second sample data to obtain the target model corresponding to the second layer.

[0150] In this embodiment, the second sample data can be input into the target model corresponding to the first layer to obtain the output result of the target model corresponding to the first layer, and the output result of the target model corresponding to the first layer can be used as the second label data; the second sample data can be input into the second model corresponding to the second layer to obtain the output result of the second model corresponding to the second layer, and then a loss function can be used to calculate the loss value based on the second label data and the output result of the second model corresponding to the second layer, and the model parameters of the second model corresponding to the second layer can be updated based on the loss value to obtain the target model corresponding to the second layer.

[0151] Step c3: Process the second sample data using the target model corresponding to the second layer to obtain the third label data. Based on the third label data and the second sample data, perform a second training on the second model corresponding to the third layer to obtain the target model corresponding to the third layer. Continue until the second model corresponding to the Nth layer is trained a second time based on the Nth label data and the second sample data to obtain the target model corresponding to the Nth layer.

[0152] In this embodiment, the second sample data can be input into the target model corresponding to the second layer, allowing the target model to process the second sample data. The output of the target model is then used as the third label data. The second sample data is then input into the second model corresponding to the third layer, yielding the output of the second model. A loss function is then used to calculate the loss value based on the third label data and the output of the second model. Based on the calculated loss value, the model parameters of the second model are updated to obtain the target model for the third layer. This process continues until a loss function is used to calculate the loss value based on the Nth label data and the output of the second model corresponding to the Nth layer. The model parameters of the second model corresponding to the Nth layer are then updated based on this loss value to obtain the target model for the Nth layer. The Nth label data can be obtained by processing the second sample data using the target model corresponding to the (N-1)th layer.

[0153] The following describes in detail the information processing method provided in the embodiments of this application, in conjunction with application scenarios.

[0154] In one feasible implementation, there can be 3 target devices, each with a different number of second parameters. The number of second parameters corresponding to the 3 target devices can be sorted from largest to smallest to obtain a sorted number of second parameters. Based on the sorted number of second parameters, the number of second parameters corresponding to each layer is determined. The number of layers in the layered compression is 3.

[0155] In this configuration, the number of second parameters corresponding to the first layer is greater than that corresponding to the second layer, and the number of second parameters corresponding to the second layer is greater than that corresponding to the third layer. The number of layers is the same as the number of target devices, which is 3. This means it's a three-layer compression process, also known as three-stage compression.

[0156] The following section provides a detailed explanation of the parallel layered compression in conjunction with Tables 1 and 2.

[0157] In this embodiment, parallel layered compression primarily involves initiating compression of the current layer before the previous layer has finished compressing, meaning that both the previous and current layers can be compressed within a certain timeframe. This can also be understood as parallel compression starting compression at each layer in a pipelined manner, allowing multiple layers to compress simultaneously. These layers are interconnected, progressing sequentially from large to small, and the change is gradual and incremental. The advantage of this approach is that adjacent layers exhibit less variation during compression, have a higher degree of correlation, and are easier to transfer knowledge.

[0158] In one feasible implementation, the initial model can be the original self-supervised model S0. The first model corresponding to the first layer can be a low-compression model S1, the first model corresponding to the second layer can be a medium-compression model S2, and the first model corresponding to the third layer can be a high-compression model S3. After training the original self-supervised model S0 for one epoch, compression of the first layer begins in the second epoch. The compression intensity of each layer is determined by the pruning ratio of the first layer. This pruning ratio can be referred to as the pruning parameter increment strategy. Table 1 below shows the first model corresponding to each layer of parallel layered compression, the number of second parameters for each layer, the pruning ratio for each layer, and the number of dynamic parameters remaining in real-time for each layer. The number of dynamic parameters remaining in real-time for each layer corresponds to the number of third parameters for each layer.

[0159] In Table 2, the cutting ratios for each layer are the same, but they can also be set to be different.

[0160]

[0161] Table 1

[0162] As shown in Table 1, the current clipping ratio of the first layer increases by 5% compared to the previous clipping ratio of the first layer; the number of second parameters corresponding to the first layer can be understood as the number of second parameters of the low-compression model S1, specifically 80; the number of parameter changes between two consecutive compressions of the first layer is (100-80)*5%=1; the arrows in Table 2 indicate that the number of third parameters corresponding to each compression of each layer is calculated based on the number of third parameters corresponding to the previous layer. N0' is always 100.

[0163] As shown in Table 2, in the second epoch, the first compression of the first layer is initiated, and the number of third parameters corresponding to the first compression of the first layer is N1' = 99. In the third epoch, the second compression of the first layer is performed, and the number of third parameters corresponding to the second compression of the first layer is N1' = 98. In the fourth epoch, the third compression of the first layer is performed, and the number of third parameters corresponding to the third compression of the first layer is N1' = 97. The training phases in Table 2 include the first training and the second training; training can also occur between the first and second training phases.

[0164]

[0165] Table 2

[0166] As shown in Table 2, the second layer of compression begins in the third epoch. The number of third parameters corresponding to each compression in the second layer can be N2' = N1' - (N1' - 60) * P2. When calculating N2', the value of (N1' - 60) * P2 needs to be rounded up, and N1' is then rounded up to the value obtained. The number of third parameters corresponding to the first compression in the second layer can be N2' = 99 - (99 - 60) * 5% = 99 - 2 = 97; the number of third parameters corresponding to the second compression in the second layer can be N2' = 98 - (98 - 60) * 10% = 98 - 4 = 94; and the number of third parameters corresponding to the third compression in the second layer can be N2' = 97 - (97 - 60) * 15% = 97 - 6 = 91.

[0167] As shown in Table 2, the third layer of compression begins at the 4th epoch. The number of third parameters corresponding to each compression in the third layer can be N3' = N2' - (N2' - 40) * P3. When calculating N3', the value of (N2' - 40) * P3 needs to be rounded up, and N2' is subtracted from the rounded-up value. Specifically, the number of third parameters corresponding to the first compression in the third layer can be N3' = 97 - (97 - 40) * 5% = 97 - 3 = 94; the number of third parameters corresponding to the second compression in the third layer can be N3' = 94 - (94 - 40) * 10% = 94 - 6 = 88; and the number of third parameters corresponding to the third compression in the third layer can be N3' = 91 - (91 - 40) * 15% = 91 - 8 = 83. N1' can also be understood as the number of weight parameters remaining after the first compression; N2' can be understood as the number of weight parameters remaining after the second compression; and N3' can be understood as the number of weight parameters remaining after the third compression.

[0168] The following section provides a detailed explanation of the serial layered compression in conjunction with Tables 3 and 4.

[0169] In the embodiments of this application, the serial layered compression is mainly manifested in that the second layer of compression is started only after the first layer of compression is completed and S1 is obtained, and the third layer of compression is started only after the second layer of compression is completed and S2 is obtained.

[0170] Table 3 below shows the first model for each layer of serial layered compression, the number of second parameters for each layer, the pruning ratio for each layer, and the number of dynamic parameters remaining in real-time for each layer. Note that the pruning ratio for each layer in Table 3 is different; however, it can also be set to be the same.

[0171]

[0172] Table 3

[0173] As shown in Table 4, it can be seen that the second layer of compression is started only when N1' = 80, that is, when the first layer of compression ends and S1 is obtained, and the third layer of compression is started only when the second layer of compression ends and S2 is obtained.

[0174]

[0175] Table 4

[0176] The training phases in Table 4 include the first training and the second training, or the training phase between the first training and the second training.

[0177] It should be noted that, whether it is serial hierarchical compression or parallel hierarchical compression, each compression in each layer includes two processes. The first process is the selection of model parameters (i.e., the process of obtaining the labeled sub-model), and the second process is the updating of model parameters (i.e., the process of updating the labeled sub-model).

[0178] Taking the current layer as an example, in each compression of the current layer, the model parameter selection refers to filtering the model parameters of the compressed sub-model obtained from the previous compression of the current layer according to their importance, obtaining the weight parameters corresponding to the number of third parameters for the current layer in this compression, and labeling the weight parameters of the compressed sub-model obtained from the previous compression of the current layer (excluding the weight parameters corresponding to the number of third parameters for the current layer in this compression) with invalid labels, thus obtaining the labeled sub-model for the current layer in this compression. Then, the model parameters of the labeled sub-model for the current layer in this compression can be updated based on unlabeled data to obtain the compressed sub-model corresponding to the current layer in this compression. It is important to note that, regardless of whether it is serial or parallel layered compression, there is no direct relationship between the model parameters selected in the previous compression of the current layer and the model parameters selected in the current compression. For example, the 90 weight parameters obtained from the first compression of the first layer and the 80 weight parameters obtained from the second compression of the first layer in Table 4 are not obtained by selecting the 90 weight parameters obtained from the previous compression, but rather by selecting the 100 weight parameters of the compressed sub-model obtained after the first compression of the first layer.

[0179] It should be noted that when updating the model parameters of the labeled sub-model of the current layer based on unlabeled sample data, the unlabeled sample data can be input into the labeled sub-model of the current layer to obtain its output, and then input into S0 to obtain its output. A loss function is then used to calculate the loss value based on the outputs of the labeled sub-model and S0, and this loss value is used to update the values ​​of the 100 weight parameters in the labeled sub-model, resulting in the compressed sub-model corresponding to this compression. Specifically, when the labeled sub-model of the current layer processes unlabeled sample data, weight parameters with invalid labels do not participate in the calculation of the unlabeled sample data. However, when updating the weight parameters of the labeled sub-model of the current layer, all weight parameters need to be updated. In other words, weight parameters with invalid labels do not participate in the forward data calculation but participate in the backward gradient update.

[0180] The following combination Figure 4 The information processing method provided in this application embodiment will be explained in detail.

[0181] like Figure 4 As shown, a pipelined, progressively increasing search function can be used to perform pipelined layered compression based on a large amount of unlabeled sample data, a small amount of labeled sample data, the hardware configuration information of each target device, the number of first parameters of S0, and the pruning ratio of each layer. This yields the first model S1 for the first layer, the second model S2 for the second layer, and the third model S3 for the third layer. Then, a large amount of unlabeled sample data can be input into S0 to obtain its output. The output of S0 is then used as the first labeled data. Based on the first labeled data and a large amount of unlabeled data, the first model for each layer is trained (i.e., self-distilled) to obtain the second model for each layer. Alternatively, the first model for each layer can be trained (i.e., self-distilled) based on the first labeled data, a large amount of unlabeled data, and a small amount of labeled data to obtain the second model for each layer. The second model corresponding to the first layer can be used as the target model for the first layer. Based on a large amount of unlabeled data and the target model corresponding to the first layer, the second model corresponding to the second layer is trained under guided training (i.e., co-distillation) to obtain the target model corresponding to the second layer. This process continues until the second model corresponding to the Nth layer is trained under guided training based on a large amount of unlabeled data and the target model corresponding to the N-1th layer to obtain the target model corresponding to the Nth layer. Then, the target model corresponding to each layer is deployed on the target device corresponding to that layer to meet deployment requirements. Here, unlabeled sample data can be called unlabeled data, and labeled sample data can be called labeled data.

[0182] It's important to note that a relatively high-precision initial model is obtained by training a large model on a self-supervised scenario with limited labeled data. Then, a pipelined, layered compression process is performed using subsequent unlabeled sample data, ultimately resulting in a higher-precision first model (small model) for each layer. The initial model can be used to process a large amount of unlabeled data, with its output serving as soft labels. Model parameters are updated in each compression cycle based on these soft labels. This allows for relatively direct transfer of knowledge from the large model to the compressed small models via soft labels. Furthermore, because the small models are obtained through pipelined compression, the overall gap between the two models is smaller, facilitating knowledge transfer and making it easier to find more suitable small models. Moreover, since the entire process is incrementally pipelined, the knowledge transfer in the compressed small models is more robust, resulting in stronger robustness and generalization. When the number of layers in the hierarchical compression is 3, the pipelined hierarchical compression can be understood as follows: first, a small amount of compression is performed based on the knowledge transfer from the initial model to obtain the first-stage search compression target; then, based on the compression target obtained in the first stage, the second-stage search compression is performed based on unlabeled sample data; finally, based on the compression target obtained in the second stage, the third-stage transfer search is started based on unlabeled data.

[0183] Thus, by performing layered compression on the initial model, a first model corresponding to each layer is obtained, reducing the training time and difficulty for small models to achieve high accuracy in the self-supervised domain. It makes full use of the soft labels of the original self-supervised model in scenarios with a large amount of unlabeled data, improving the accuracy of small models during pipelined compression and training, and gradually increasing the compression intensity of the model. This improves the compression intensity while maintaining the overall accuracy of the small models, thereby increasing their robustness in deployment and operation (pipeline stage). This solution obtains multiple high-precision small models in one training process, accelerating the progress and quality of model deployment.

[0184] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0185] The information processing method provided in the embodiments of this application uses a pipelined approach to perform layered compression of the initial model. The first model corresponding to each layer retains the knowledge of the initial model. Furthermore, when using a pipelined approach for layered compression, the compression span is small, making it easier to transfer the knowledge of the initial model to the first model corresponding to each layer. Unlike related technologies, it no longer relies on a small amount of labeled sample data for model training to obtain the first model, thus improving the accuracy of the obtained first model. This solves the problem in related technologies where the small models trained have low accuracy and are difficult to meet deployment requirements.

[0186] Based on the foregoing embodiments, embodiments of this application provide an information processing apparatus that can be applied to... Figures 1-3 In the information processing method provided in the corresponding embodiment, refer to Figure 5 As shown, the information processing device 4 includes:

[0187] Acquisition unit 41 is used to acquire the initial model and acquire the hardware configuration information of multiple target devices;

[0188] Processing unit 42 is used to determine the second parameter information corresponding to each target device based on the first parameter information and hardware configuration information of the initial model;

[0189] The processing unit 42 is also used to perform layered compression of the initial model in a pipeline manner based on multiple second parameter information and first parameter information to obtain the first model corresponding to each layer; wherein, the first model corresponding to the current layer is generated based on the model parameters corresponding to the previous layer.

[0190] In other embodiments of this application, the processing unit 42 is specifically used to perform the following steps:

[0191] Obtain the clipping and compression relationship for each layer; where the clipping and compression relationship is the correspondence between the number of compressions and the clipping ratio;

[0192] Based on the number of first parameters of the initial model, the number of second parameters corresponding to each layer, and the pruning and compression relationship, the number of third parameters corresponding to each layer of compression is determined; wherein, the first parameter information includes the number of first parameters; the second parameter information includes the number of second parameters;

[0193] Based on the number of the third parameter, the initial model is compressed in layers using a pipeline approach to obtain the first model corresponding to each layer.

[0194] In other embodiments of this application, the processing unit 42 is specifically used to perform the following steps:

[0195] Based on the number of third parameters corresponding to the first layer of compression, the initial model is periodically compressed to obtain the first model corresponding to the first layer.

[0196] In the process of periodically compressing the initial model to obtain the first model corresponding to the first layer, based on the number of third parameters corresponding to the second layer compression, the first target compressed sub-model corresponding to the first layer is periodically compressed to obtain the first model corresponding to the second layer, until the process of obtaining the first model corresponding to the (N-1)th layer is periodically compressed to obtain the first model corresponding to the Nth layer; wherein, the first target compressed sub-model is obtained by compressing the initial model; or,

[0197] Based on the number of third parameters corresponding to the second layer of compression, the first model corresponding to the first layer is periodically compressed to obtain the first model corresponding to the second layer, until the first model corresponding to the (N-1)th layer is compressed to obtain the first model corresponding to the Nth layer; where N is a positive integer greater than 1.

[0198] In other embodiments of this application, the processing unit 42 is specifically used to perform the following steps:

[0199] Given that the model parameters of the initial model have been updated for the Mth time to obtain the Mth updated model, the Mth updated model is compressed for the first time based on the number of third parameters corresponding to the first compression of the first layer to obtain the first first compressed sub-model corresponding to the first layer. At the same time, the model parameters of the Mth updated model are updated for the M+1th time to obtain the M+1th updated model.

[0200] Based on the number of third parameters corresponding to the second compression of the second layer and the M+1th update model, the first first compression sub-model is compressed a second time to obtain the second first compression sub-model corresponding to the first layer. At the same time, the M+1th update model is updated for the M+2th time to obtain the M+2th update model. This process continues until the number of parameters of the i-th first compression sub-model corresponding to the first layer satisfies the number of second parameters corresponding to the first layer. The i-th first compression sub-model is then used as the first model. Here, M and i are positive integers.

[0201] In other embodiments of this application, the processing unit 42 is specifically used to perform the following steps:

[0202] Based on the number of third parameters corresponding to the first compression of the first layer, the initial model is compressed for the first time to obtain the first compressed sub-model corresponding to the first layer.

[0203] Based on the number of third parameters corresponding to the second compression of the first layer and the initial model, the first first compression sub-model is compressed a second time to obtain the second first compression sub-model corresponding to the first layer, until the number of parameters of the i-th first compression sub-model satisfies the number of second parameters corresponding to the first layer, and the i-th first compression sub-model is taken as the first model corresponding to the first layer; where i is a positive integer.

[0204] In other embodiments of this application, the processing unit 42 is specifically used to perform the following steps:

[0205] Given that the initial model has been compressed for the jth time to obtain the jth first compressed sub-model corresponding to the first layer, the jth first compressed sub-model is compressed for the first time based on the number of third parameters corresponding to the first compression of the second layer to obtain the 1st second compressed sub-model corresponding to the second layer. At the same time, the initial model is compressed for the (j+1)th time to obtain the (j+1)th first compressed sub-model.

[0206] Based on the number of third parameters corresponding to the second compression of the second layer and the (j+1)th first compression sub-model, the first second compression sub-model is compressed a second time to obtain the second second compression sub-model corresponding to the second layer. At the same time, the initial model is compressed j+2 times to obtain the (j+2)th first compression sub-model. This process continues until the number of parameters of the obtained h-th second compression sub-model satisfies the number of second parameters corresponding to the second layer. The h-th second compression sub-model is then used as the first model corresponding to the second layer. Here, h and j are both positive integers.

[0207] In other embodiments of this application, the processing unit 42 is specifically used to perform the following steps:

[0208] Based on the number of third parameters corresponding to the first compression of the first layer, the initial model is compressed for the first time to obtain the initial compressed sub-model;

[0209] The model parameters of the initial compression sub-model are updated to obtain the first compression sub-model.

[0210] In other embodiments of this application, the processing unit 42 is further configured to perform the following steps:

[0211] Obtain the first sample data and process it using the initial model to obtain the first label data;

[0212] Based on the first sample data and the first label data, the first model corresponding to each layer is trained to obtain the second model corresponding to each layer.

[0213] The second model corresponding to each layer is trained a second time to obtain the target model corresponding to each layer.

[0214] In other embodiments of this application, the processing unit 42 is further configured to perform the following steps:

[0215] Obtain the second sample data and determine the second model corresponding to the first layer as the target model corresponding to the first layer;

[0216] The target model corresponding to the first layer is used to process the second sample data to obtain the second label data. Based on the second label data and the second sample data, the second model corresponding to the second layer is trained again to obtain the target model corresponding to the second layer.

[0217] The second sample data is processed using the target model corresponding to the second layer to obtain the third label data. Based on the third label data and the second sample data, the second model corresponding to the third layer is trained a second time to obtain the target model corresponding to the third layer. This process continues until the second model corresponding to the Nth layer is trained a second time based on the Nth label data and the second sample data to obtain the target model corresponding to the Nth layer.

[0218] It should be noted that the interaction process between the units in the embodiments of this application can be referred to Figures 1-3 The implementation process of the information processing method provided in the corresponding embodiments will not be described in detail here.

[0219] The information processing apparatus provided in this application uses a pipelined approach to perform layered compression of the initial model. The first model corresponding to each layer retains the knowledge of the initial model. Furthermore, when using a pipelined approach for layered compression, the compression span is small, making it easier to transfer the knowledge of the initial model to the first model corresponding to each layer. Unlike related technologies, it no longer relies on a small amount of labeled sample data for model training to obtain the first model, thus improving the accuracy of the obtained first model. This solves the problem in related technologies where the small models trained have low accuracy and are difficult to meet deployment requirements.

[0220] Based on the foregoing embodiments, embodiments of this application provide an electronic device that can be applied to... Figures 1-3 In the information processing method provided in the corresponding embodiment, refer to Figure 6 As shown, the electronic device 5 may include: a processor 51, a memory 52, and a communication bus 53, wherein:

[0221] Communication bus 53 is used to realize the communication connection between processor 51 and memory 52;

[0222] The processor 51 is used to execute the information processing program stored in the memory 52 to achieve the above. Figures 1-3 The corresponding embodiments provide the steps of the information processing method.

[0223] It should be noted that the specific implementation process of the steps executed by the processor in this embodiment can be referred to the implementation process of the information processing method provided in the embodiments corresponding to 1 to 3, and will not be repeated here.

[0224] The electronic device provided in the embodiments of this application uses a pipelined approach to perform layered compression of the initial model. The first model corresponding to each layer retains the knowledge of the initial model. Furthermore, when using a pipelined approach for layered compression, the compression span is small, making it easier to transfer the knowledge of the initial model to the first model corresponding to each layer. Unlike related technologies, it no longer relies on a small amount of labeled sample data for model training to obtain the first model, thus improving the accuracy of the obtained first model. This solves the problem in related technologies where the small models trained have low accuracy and are difficult to meet deployment requirements.

[0225] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement... Figures 1-3 The steps in the information processing method provided in the corresponding embodiment.

[0226] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0227] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0228] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0229] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0230] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0231] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0232] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0233] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An information processing method, wherein, The method includes: Obtain the initial model and the hardware configuration information of multiple target devices; Based on the first parameter information of the initial model and the hardware configuration information, the second parameter information corresponding to each target device is determined; Based on multiple second parameter information and the first parameter information, the initial model is compressed in a layered manner using a pipelined approach to obtain a first model corresponding to each layer. A target model is deployed on the target device corresponding to each layer, and the target model is obtained based on the first model. The first model corresponding to the current layer is generated based on the model parameters of the previous layer; the second parameter information includes the number of model parameters of the model that the target device can run for each layer; the pipelined mode represents the sequential activation of compression for each layer, and the compression of the current layer depends on the compression of the previous layer; the layered compression includes parallel layered compression; the number of target devices is the same as the number of layers in the layered compression.

2. The method of claim 1, wherein, The step of performing layered compression on the initial model using a pipelined approach based on multiple second parameter information and the first parameter information to obtain a first model corresponding to each layer includes: Obtain the clipping and compression relationship corresponding to each layer; wherein, the clipping and compression relationship is the correspondence between the number of compressions and the clipping ratio; Based on the number of first parameters of the initial model, the number of second parameters corresponding to each layer, and the pruning and compression relationship, the number of third parameters corresponding to each layer compression is determined; wherein, the first parameter information includes the number of first parameters; and the second parameter information includes the number of second parameters. Based on the number of the third parameter, the initial model is compressed in a layered manner using a pipeline method to obtain the first model corresponding to each layer.

3. The method of claim 2, wherein, Based on the number of the third parameter, the initial model is compressed in a layered manner using a pipeline approach to obtain the first model corresponding to each layer, including: Based on the number of third parameters corresponding to the first layer of compression, the initial model is periodically compressed to obtain the first model corresponding to the first layer; In the process of periodically compressing the initial model to obtain the first model corresponding to the first layer, based on the number of third parameters corresponding to the second layer compression, the first target compression sub-model corresponding to the first layer is periodically compressed to obtain the first model corresponding to the second layer, until the process of obtaining the first model corresponding to the (N-1)th layer, the (N-1)th target compression sub-model corresponding to the (N-1)th layer is periodically compressed to obtain the first model corresponding to the Nth layer; wherein, the first target compression sub-model is obtained by compressing the initial model; wherein, N is a positive integer greater than 1.

4. The method of claim 3, wherein, The step of periodically compressing the initial model based on the number of third parameters corresponding to the first layer of compression to obtain the first model corresponding to the first layer includes: When the model parameters of the initial model are updated for the Mth time to obtain the Mth updated model, the Mth updated model is compressed for the first time based on the number of third parameters corresponding to the first compression of the first layer to obtain the first first compressed sub-model corresponding to the first layer. At the same time, the model parameters of the Mth updated model are updated for the M+1th time to obtain the M+1th updated model. Based on the number of third parameters corresponding to the second compression of the first layer and the (M+1)th update model, the first first compression sub-model is compressed a second time to obtain the second first compression sub-model corresponding to the first layer. At the same time, the (M+1)th update model is updated for the (M+2)th time to obtain the (M+2)th update model. This process continues until the number of parameters of the i-th first compression sub-model corresponding to the first layer satisfies the number of second parameters corresponding to the first layer. The i-th first compression sub-model is then used as the first model. Here, M and i are positive integers.

5. The method of claim 3, wherein, The step of periodically compressing the initial model based on the number of third parameters corresponding to the first layer of compression to obtain the first model corresponding to the first layer includes: Based on the number of third parameters corresponding to the first compression of the first layer, the initial model is compressed for the first time to obtain the first compressed sub-model corresponding to the first layer. Based on the number of third parameters corresponding to the second compression of the first layer and the initial model, the first first compression sub-model is compressed a second time to obtain the second first compression sub-model corresponding to the first layer, until the number of parameters of the i-th first compression sub-model satisfies the number of second parameters corresponding to the first layer, and the i-th first compression sub-model is taken as the first model corresponding to the first layer; where i is a positive integer.

6. The method of claim 3, wherein, In the process of periodically compressing the initial model to obtain the first model corresponding to the first layer, based on the number of third parameters corresponding to the second layer compression, periodically compressing the first target compressed sub-model corresponding to the first layer to obtain the first model corresponding to the second layer includes: When the initial model is compressed for the jth time to obtain the jth first compressed sub-model corresponding to the first layer, the jth first compressed sub-model is compressed for the first time based on the number of third parameters corresponding to the first compression of the second layer to obtain the 1st second compressed sub-model corresponding to the second layer. At the same time, the initial model is compressed for the (j+1)th time to obtain the (j+1)th first compressed sub-model. Based on the number of third parameters corresponding to the second compression of the second layer and the (j+1)th first compression sub-model, the first second compression sub-model is compressed a second time to obtain the second second compression sub-model corresponding to the second layer. At the same time, the initial model is compressed j+2 times to obtain the (j+2)th first compression sub-model. This process continues until the number of parameters of the obtained h-th second compression sub-model satisfies the number of second parameters corresponding to the second layer. The h-th second compression sub-model is then used as the first model corresponding to the second layer. Here, h and j are both positive integers.

7. The method of claim 5, wherein, The step of performing a first compression on the initial model based on the number of third parameters corresponding to the first compression of the first layer to obtain the first compressed sub-model corresponding to the first layer includes: Based on the number of third parameters corresponding to the first compression of the first layer, the initial model is compressed for the first time to obtain the initial compressed sub-model; The model parameters of the initial compression sub-model are updated to obtain the first compression sub-model.

8. The method of claim 1, wherein, The method further includes: Obtain first sample data and process the first sample data using the initial model to obtain first label data; Based on the first sample data and the first label data, the first model corresponding to each layer is trained to obtain the second model corresponding to each layer. The second model corresponding to each layer is trained a second time to obtain the target model corresponding to each layer.

9. The method of claim 8, wherein, The step of performing a second training on the second model corresponding to each layer to obtain the target model corresponding to each layer includes: Obtain the second sample data and determine the second model corresponding to the first layer as the target model corresponding to the first layer; The second sample data is processed using the target model corresponding to the first layer to obtain the second label data. Based on the second label data and the second sample data, the second model corresponding to the second layer is trained again to obtain the target model corresponding to the second layer. The second sample data is processed using the target model corresponding to the second layer to obtain the third label data. Based on the third label data and the second sample data, the second model corresponding to the third layer is trained a second time to obtain the target model corresponding to the third layer. This process continues until the second model corresponding to the Nth layer is trained a second time based on the Nth label data and the second sample data to obtain the target model corresponding to the Nth layer.

10. An information processing apparatus, comprising: The device includes: The acquisition unit is used to acquire the initial model and the hardware configuration information of multiple target devices; The processing unit is used to determine the second parameter information corresponding to each target device based on the first parameter information of the initial model and the hardware configuration information; The processing unit is further configured to perform layered compression on the initial model in a pipelined manner based on multiple second parameter information and the first parameter information, to obtain a first model corresponding to each layer. A target model is deployed on the target device corresponding to each layer, and the target model is obtained based on the first model. The first model corresponding to the current layer is generated based on the model parameters corresponding to the previous layer. The second parameter information includes the number of model parameters of the model that the target device corresponding to each layer can run. The pipelined manner represents the sequential activation of compression of each layer, and the compression of the current layer depends on the compression of the previous layer. The layered compression includes parallel layered compression. The number of target devices is the same as the number of layers in the layered compression.

Images