Model pruning method and device, electronic equipment and storage medium

Abstract

The invention provides a model pruning method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-pruned model and a plurality of input data; executing the following processing through the target sub-model: predicting input data corresponding to the target sub-model to obtain a prediction result of the target sub-model; determining the distance between the input data corresponding to the target sub-model and other input data; based on the prediction result of each sub-model, performing weight prediction on the target sub-model to obtain a prediction weight of the target sub-model, and fusing the distance and the prediction weight to obtain a target weight of the target sub-model; and performing sub-model pruning on the to-be-pruned model based on the target weight of each sub-model to obtain a pruned model. According to the invention, the pruning accuracy of the sub-model can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a model pruning method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of deep learning technology, model complexity and computing requirements continue to increase, which makes it difficult to deploy deep learning models on resource-constrained devices. Model pruning, as an effective model compression and optimization method, aims to reduce the amount of computation and storage requirements by removing unimportant parts of the model while maintaining the model's predictive performance. However, most of the related model pruning methods are based on simple gradient information for pruning, which leads to a decrease in the performance of the pruned model. Summary of the invention

[0003] The embodiments of the present application provide a model pruning method, device, electronic device and storage medium, which can improve the accuracy of sub-model pruning.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a model pruning method, the method comprising:

[0006] Acquire a model to be pruned and a plurality of input data, wherein the model to be pruned includes a plurality of sub-models, and the input data corresponds to the sub-models one by one;

[0007] The following processing is performed by the target sub-model: predicting the input data corresponding to the target sub-model to obtain a prediction result of the target sub-model;

[0008] Determine a distance between the input data corresponding to the target sub-model and other input data, wherein the other input data is the input data other than the input data corresponding to the target sub-model among the multiple input data;

[0009] Based on the prediction results of each of the sub-models, weight prediction is performed on the target sub-model to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are fused to obtain the target weight of the target sub-model;

[0010] Based on the target weight of each sub-model, sub-model pruning is performed on the model to be pruned to obtain a pruned model.

[0011] The present application provides a model pruning device, including:

[0012] A data acquisition module, used to obtain a model to be pruned and a plurality of input data, wherein the model to be pruned includes a plurality of sub-models, and the input data corresponds to the sub-models one by one;

[0013] A prediction module is used to perform the following processing through the target sub-model: predict the input data corresponding to the target sub-model to obtain the prediction result of the target sub-model;

[0014] A pruning module is used to determine the distance between the input data corresponding to the target sub-model and other input data, wherein the other input data is the input data other than the input data corresponding to the target sub-model among the multiple input data; based on the prediction result of each of the sub-models, weight prediction is performed on the target sub-model to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are integrated to obtain the target weight of the target sub-model; based on the target weight of each of the sub-models, sub-model pruning is performed on the model to be pruned to obtain a pruned model.

[0015] An embodiment of the present application provides an electronic device, including:

[0016] A memory for storing computer executable instructions;

[0017] The processor is used to implement the model pruning method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the model pruning method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the model pruning method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] The input data corresponding to the target sub-model is predicted to obtain the prediction result of the target sub-model, the distance between the input data corresponding to the target sub-model and other input data is determined, and based on the prediction result of each sub-model, the target sub-model is weighted predicted to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are fused to obtain the target weight of the target sub-model. Based on the target weight of each sub-model, the model to be pruned is sub-model pruned to obtain the pruned model. In this way, the distance between the input data of the model and the prediction ability of the model are comprehensively considered when pruning the sub-model, which improves the accuracy of sub-model pruning and ensures the performance of the pruned model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the architecture of the model pruning system provided in an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 3A It is a first flow chart of the model pruning method provided in an embodiment of the present application;

[0025] Figure 3B It is a second flow chart of the model pruning method provided in an embodiment of the present application;

[0026] Figure 3C It is a third flow chart of the model pruning method provided in the embodiment of the present application;

[0027] Figure 3D It is a fourth flow chart of the model pruning method provided in an embodiment of the present application;

[0028] Figure 3E It is a fifth flow chart of the model pruning method provided in the embodiment of the present application;

[0029] Figure 3F It is a sixth flow chart of the model pruning method provided in the embodiment of the present application;

[0030] Figure 3G It is a seventh flow chart of the model pruning method provided in the embodiment of the present application;

[0031] Figure 4 It is a schematic diagram of the network model structure provided in the embodiment of the present application;

[0032] Figure 5 It is a schematic diagram of the sub-network model structure provided in an embodiment of the present application;

[0033] Fig. 6A is a first data processing flow chart provided in an embodiment of the present application;

[0034] Figure 6B is a second data processing flow chart provided in an embodiment of the present application;

[0035] Figure 6C is a third data processing flow chart provided in an embodiment of the present application;

[0036] Figure 7 It is a schematic diagram of the interaction interface between a user and a product provided in an embodiment of the present application;

[0037] Figure 8 It is a first feature importance schematic diagram provided in an embodiment of the present application;

[0038] Fig. 9A This is a first schematic diagram of the relationship between age and purchase conversion rate provided in an embodiment of the present application;

[0039] Fig. 9B This is a first schematic diagram of the distribution density of ages provided in an embodiment of the present application;

[0040] Fig. 9C is a schematic diagram of the relationship between the number of held products and the purchase conversion rate provided in an embodiment of the present application;

[0041] Fig.9D This is a schematic diagram of the distribution density of the number of held products provided in an embodiment of the present application;

[0042] Fig.10 It is a schematic diagram of the interaction interface between a user and a traffic data package provided in an embodiment of the present application;

[0043] Fig.11 is a schematic diagram of the importance of the second feature provided in an embodiment of the present application;

[0044] Fig. 12A This is a schematic diagram of the relationship between the one-day conversion rate and the purchase conversion rate provided in the embodiment of the present application;

[0045] Fig. 12B This is a schematic diagram of the distribution density of the one-day conversion rate provided in the embodiment of the present application;

[0046] Fig. 12C It is a schematic diagram of the relationship between the click conversion rate and the purchase conversion rate provided in the embodiment of the present application;

[0047] Fig.12D It is a schematic diagram of the distribution density of click-through conversion rate provided in an embodiment of the present application;

[0048] Fig.12E is a second schematic diagram of the relationship between age and purchase conversion rate provided in an embodiment of the present application;

[0049] Fig.12F This is a second schematic diagram of the distribution density of ages provided in an embodiment of the present application.

[0050] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0052] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0054] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0055] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0056] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0057] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0058] 1) Input data is feature data determined based on initial data. Initial data is data that has not been preprocessed. Each initial data has a corresponding category, and different initial data have different categories. The embodiment of the present application does not limit the category of initial data. The category of initial data can be user attributes, product attributes, interactive content, product identification, etc. User attributes are information describing users, which are used to divide users into fine-grained categories, such as: statistical information such as age, gender, and education level, geographical location information such as residential city and residential area, and psychological characteristic information such as interests and preferences; product attributes are information describing products (i.e., applications), which are used to help users understand the functions and quality of products, such as: product size, color, weight, function, use, etc.; interactive content is information generated when users interact with products or services, which is used to reflect user preferences and satisfaction, such as: feedback information such as evaluations and comments, usage information such as frequency of use, usage time, and usage scenarios, and behavioral records such as number of clicks, browsing time, search content, and purchase time; product identification is information that uniquely identifies a product, which is used to identify the product, such as a universal product code or an international product number.

[0059] In the related art, pruning is performed based on simple weight size or gradient information, which ignores the feature interaction between input data and the difference in prediction performance of the model, reduces the accuracy of sub-model pruning, and causes the performance of the pruned model to deteriorate. To address the above problems, the embodiments of the present application provide a model pruning method, device, electronic device, computer-readable storage medium and computer program product to improve the accuracy of sub-model pruning.

[0060] The model pruning method described in the embodiments of the present application can be applied to various fields, for example, multiple fields such as recommendation fields and image recognition fields, that is, the model pruning method in the embodiments of the present application is not limited to a certain field.

[0061] The following describes an exemplary application of the electronic device provided in the embodiment of the present application. The device provided in the embodiment of the present application can be implemented as a terminal or a server. The following describes an exemplary application when the device is implemented as a server.

[0062] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the model pruning system 100 provided in an embodiment of the present application. To support a model pruning application, a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0063] The terminal 400 is used to send the model to be pruned, including multiple sub-models, and multiple input data to the server 200 through the network 300. The server 200 is used to perform the following processing through the target sub-model: predict the input data corresponding to the target sub-model to obtain the prediction result of the target sub-model; determine the distance between the input data corresponding to the target sub-model and other input data; based on the prediction result of each sub-model, perform weight prediction on the target sub-model to obtain the prediction weight of the target sub-model, and fuse the distance and the prediction weight to obtain the target weight of the target sub-model; based on the target weight of each sub-model, perform sub-model pruning on the model to be pruned to obtain the pruned model, and return the pruned model to the terminal 400. The terminal 400 displays the pruned model through the graphical interface 410.

[0064] An example of terminal 400 performing model pruning is described below.

[0065] In some embodiments, terminal 400 can independently complete the model pruning task. For example, terminal 400 is used to obtain a model to be pruned including multiple sub-models and multiple input data; perform the following processing through the target sub-model: predict the input data corresponding to the target sub-model to obtain the prediction result of the target sub-model; determine the distance between the input data corresponding to the target sub-model and other input data; based on the prediction result of each sub-model, perform weight prediction on the target sub-model to obtain the prediction weight of the target sub-model, and fuse the distance and the prediction weight to obtain the target weight of the target sub-model; based on the target weight of each sub-model, perform sub-model pruning on the model to be pruned to obtain the pruned model, and display the pruned model through the graphical interface 410.

[0066] In one implementation scenario, the server or terminal can perform model pruning on the model of the recommendation field to obtain a recommendation model including multiple sub-models and multiple data to be recommended; perform the following processing through the target sub-model: predict the data to be recommended corresponding to the target sub-model to obtain the recommendation probability of the target sub-model; determine the distance between the data to be recommended corresponding to the target sub-model and other data to be recommended; based on the recommendation probability of each sub-model, perform weight prediction on the target sub-model to obtain the recommendation weight of the target sub-model, and fuse the distance and the recommendation weight to obtain the target weight of the target sub-model; based on the target weight of each sub-model, perform sub-model pruning on the recommendation model to obtain the pruned recommendation model, and the pruned recommendation model is used to predict the data to be recommended, and according to the recommendation probability of the data to be recommended corresponding to the target sub-model, the target recommendation data is screened out and pushed to the user.

[0067] In one implementation scenario, a server or terminal can perform model pruning on a model in the field of image recognition to obtain an image recognition model including multiple sub-models and multiple data to be recognized; perform the following processing through a target sub-model: recognize the data to be recognized corresponding to the target sub-model to obtain a recognition result of the target sub-model; determine the distance between the data to be recognized corresponding to the target sub-model and other data to be recognized; based on the recognition result of each sub-model, perform weight prediction on the target sub-model to obtain a recognition weight of the target sub-model, and fuse the distance and the recognition weight to obtain a target weight of the target sub-model; based on the target weight of each sub-model, perform sub-model pruning on the image recognition model to obtain a pruned image recognition model, and the pruned image recognition model is used to recognize the data to be recognized to obtain a target recognition result.

[0068] In some embodiments, server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0069] The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0070] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1 In the terminal 400 or server 200, the electronic device 500 includes: at least one processor 510, a memory 550, and at least one network interface 520. The various components in the server 200 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .

[0071] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0072] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0073] In some embodiments, when the terminal 400 independently completes the model pruning task, the server 200 provided in the embodiment of the present application does not include the user interface 530 .

[0074] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard disk drive, an optical disk drive, etc. The memory 550 may optionally include one or more storage devices that are physically located away from the processor 510.

[0075] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0076] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0077] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0078] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0079] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0080] In some embodiments, when the model pruning task is completed independently by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0081] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions; in some embodiments, when the embodiment independently completes the model pruning task by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0082] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The model pruning device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data acquisition module 5551, a prediction module 5552, a pruning module 5553 and a training module 5554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0083] It should be noted that in the following model pruning example, those skilled in the art may apply the model pruning method provided in the embodiment of the present application to prune the model to be pruned based on their understanding of the following.

[0084] See also Figure 3A , Figure 3A This is a first flow chart of the model pruning method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are used to illustrate that the model pruning method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will take the collaborative implementation by a server and a terminal as an example for explanation.

[0085] In step 101, a model to be pruned and a plurality of input data are obtained.

[0086] The model to be pruned includes multiple sub-models, and the input data corresponds to the sub-models one by one.

[0087] Here, the model to be pruned is a trained model for pruning, and the model to be pruned includes multiple sub-models. The embodiment of the present application does not limit the sub-models, and different sub-models can have the same network structure or different network structures. Each sub-model corresponds to its own input data.

[0088] It should be noted that the step of obtaining a plurality of input data is similar to the step of obtaining a plurality of first training samples and at least one second training sample in step 201 , see step 201 .

[0089] In some embodiments, see Figure 3B , Figure 3B is a second flow chart of the model pruning method provided in the embodiment of the present application, Figure 3A Before obtaining the model to be pruned in step 101, execute Figure 3B Steps 201 to 204 are described in detail below.

[0090] In step 201, a plurality of pre-trained first models and at least one pre-trained second model are obtained, and a plurality of first training samples and at least one second training sample are obtained.

[0091] The first training samples correspond one-to-one to the pre-trained first model, the second training samples correspond one-to-one to the pre-trained second model, and the second training samples are obtained by performing feature cross processing on any two first training samples.

[0092] Here, the pre-trained first model is a first model prepared for pre-training. The pre-trained first model is trained by the first training sample corresponding to the pre-trained first model to obtain the pre-trained first model. The embodiment of the present application does not limit the pre-trained first model. The pre-trained first model is used in various downstream tasks and is further fine-tuned to adapt to specific application scenarios. The pre-trained second model is used for joint training with the pre-trained first model. The joint training is used for training according to information from multiple data sources (such as the first training sample and the second training sample) to improve the generalization ability and performance of the model. The embodiment of the present application does not limit the method of obtaining multiple pre-trained first models and at least one pre-trained second model. The multiple pre-trained first models and at least one pre-trained second model can be directly obtained locally, pulled through a server, etc.

[0093] It should be noted that the first training sample is a sample feature obtained by preprocessing the initial samples (such as the first initial sample and the second initial sample).

[0094] In some embodiments, obtaining multiple first training samples in step 201 can be achieved by performing one of the following processes: obtaining multiple first initial samples, standardizing each first initial sample to obtain a standardized sample, and encoding the standardized sample to obtain a first training sample; obtaining multiple second initial samples, classifying each second initial sample to obtain a category of each second initial sample, and determining the second initial samples belonging to the same category as the first training sample.

[0095] Here, the first initial sample is a sample that has not been preprocessed, each first initial sample has a corresponding category, and different first initial samples have different categories. The embodiment of the present application does not limit the category of the first initial sample, and the category of the first initial sample can be a user attribute, a product attribute, an interactive content, a product logo, etc. The second initial sample is a sample feature obtained by preprocessing the sample that has not been preprocessed, and the second initial sample is not labeled with a corresponding category.

[0096] It should be noted that standardization is to linearly scale the data to a specific range. The embodiments of the present application do not limit the standardization method. Standardization can be normalization, standard deviation standardization, etc. Encoding is used to map the visualized data to a new feature space to obtain features for characterizing the potential structure of the first initial sample. The embodiments of the present application do not limit the encoding method. Encoding can be one-hot encoding, label encoding, etc. Classification is used to predict discrete output labels, that is, the categories of the input data, based on the input data.

[0097] Continuing from the above embodiment, the above “standardizing each first initial sample to obtain a standardized sample” can be achieved in the following way: calculating the maximum value and the minimum value among multiple first initial samples; calculating the first difference between each first initial sample and the minimum value; calculating the second difference between the maximum value and the minimum value; and determining the ratio of each first difference to the second difference as a standardized sample.

[0098] For example, calculate the maximum value (such as 5) and the minimum value (such as 1) among multiple first initial samples (such as [1, 2, 3, 4, 5]); calculate the first difference between each first initial sample and the minimum value (such as [0, 1, 2, 3, 4]); calculate the second difference between the maximum value and the minimum value (such as 4); and determine the ratio of each first difference to the second difference (such as [0, 0.25, 0.5, 0.75, 1]) as a standardized sample.

[0099] Continuing from the above embodiment, the above “encoding the standardized sample to obtain the first training sample” can be implemented in the following way: performing convolution processing on the standardized data to obtain convolution features, performing pooling processing on the convolution features to obtain pooling features, and performing mapping processing on the pooling features to obtain the first training sample.

[0100] For example, the standardized data (such as "I") is convolved to obtain convolution features (such as [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]), the convolution features are pooled to obtain pooled features (such as [0.14, 0.13, -0.06]), and the pooled features are mapped to obtain the first training sample (such as [0.52, 0.13, -0.06, 0.15]).

[0101] For example, each second initial sample (such as the sample features corresponding to "user A" are [2, 1, 3, 4, 6, 1, 3], the sample features corresponding to "product B" are [2, 1, 4, 4, 2, 1, 5], the sample features corresponding to "start browsing at 10:50" are [2, 3, 2, 2, 3, 1, 6], and the sample features corresponding to "browse for 20 minutes" are [2, 1, 2, 1, 1, 1]) is classified to obtain the category of each second initial sample (such as the category of the sample features [2, 1, 3, 4, 6, 1, 3] corresponding to "user A" is The user's attributes, the category of the sample feature [2, 1, 4, 4, 2, 1, 5] corresponding to "product B" is product attributes, the category of the sample feature [2, 3, 2, 2, 3, 1, 6] corresponding to "start browsing at 10:50" is interactive content, the category of the sample feature [2, 1, 2, 1, 1, 1, 1] corresponding to "browse for 20 minutes" is interactive content), and the second initial sample (such as [[2, 3, 2, 2, 3, 1, 6], [2, 1, 2, 1, 1, 1]]) belonging to the same category (such as interactive content) is determined as the first training sample.

[0102] In some embodiments, obtaining multiple first training samples in step 201 can also be achieved in the following way: obtaining multiple third initial samples, classifying each third initial sample to obtain the category of each third initial sample, determining the third initial samples belonging to the same category as fourth initial samples, standardizing each fourth initial sample to obtain a standardized sample, and encoding the standardized sample to obtain a first training sample.

[0103] It should be noted that the third initial sample is a sample that has not been preprocessed and has not been labeled with a category.

[0104] Continuing from the above embodiment, the above steps are similar to the above steps of "obtaining multiple first initial samples, standardizing each first initial sample to obtain a standardized sample, encoding the standardized sample to obtain a first training sample" and "obtaining multiple second initial samples, classifying each second initial sample to obtain a category of each second initial sample, and determining the second initial sample belonging to the same category as the first training sample", and will not be repeated here.

[0105] Through the embodiments of the present application, the initial data can be converted into a format that is more suitable for model processing through standardized processing and encoding, so that the model can capture the potential structure in the data and improve the accuracy of model prediction.

[0106] In step 202, for each pre-trained first model, the pre-trained first model is pre-trained using the corresponding first training sample to obtain a pre-trained first model.

[0107] In some embodiments, step 202 can be implemented in the following manner: perform the following processing on each pre-trained first model to predict the first training sample corresponding to the pre-trained first model to obtain the prediction result of the pre-trained first model; construct a loss value based on the prediction results of multiple pre-trained first models, and update the parameters of each pre-trained first model based on the loss value.

[0108] Continuing from the above embodiment, the pre-trained first model includes an encoder and a decoder. The encoder encodes the first training sample corresponding to the pre-trained first model to obtain the encoding features of the first training sample corresponding to the pre-trained first model. The decoder decodes the encoding features to obtain the prediction results of the pre-trained first model.

[0109] It should be noted that the encoder includes multiple layers of encoding layers, the input of the encoding layer of the first layer is the first training sample corresponding to the pre-trained first model, the input of each encoding layer except the first layer is the output of the previous encoding layer, the encoding feature of the first training sample corresponding to the pre-trained first model is extracted by the first encoding layer, the extracted encoding feature is normalized to obtain the output feature corresponding to the first encoding layer, the encoding feature of the output feature corresponding to the first encoding layer is extracted by the second encoding layer, multiple iterations are performed, and the output feature extracted by the last encoding layer is determined as the encoding feature of the first training sample corresponding to the pre-trained first model. The decoder includes multiple layers of decoding layers, the input of the decoding layer of the first layer is the encoding feature, the input of each decoding layer except the first layer is the output of the previous decoding layer, and the output obtained by the last decoding layer is mapped to a real number greater than or equal to 0 and less than or equal to 1, that is, the prediction result of the pre-trained first model.

[0110] In some embodiments, the above-mentioned "constructing a loss value based on the prediction results of multiple pre-trained first models" can be achieved in the following way: fusing the prediction results of multiple pre-trained first models to obtain a fused prediction result, and constructing a loss value based on the fused prediction result and the sample label of the first training sample.

[0111] Continuing from the above embodiment, the above “constructing a loss value based on the fusion prediction result and the sample label of the first training sample” can be implemented in the following way: encoding the fusion prediction result to obtain a first encoding feature; encoding the sample label of the first training sample to obtain a second encoding feature; constructing a loss value based on the first encoding feature and the second encoding feature.

[0112] It should be noted that the embodiment of the present application does not limit the method for constructing the loss value, and the loss value can be the mean absolute error, mean square error, etc. of two encoding vectors.

[0113] In some embodiments, the above-mentioned “updating the parameters of each pre-trained first model based on the loss value” can be implemented in the following manner: performing backpropagation in each pre-trained first model based on the loss value to obtain a gradient; and updating the parameters of each pre-trained first model based on the gradient.

[0114] It should be noted that back propagation is implemented through the back propagation algorithm, and the gradient of the loss value with respect to the parameters of each pre-trained first model is calculated by the derivative chain rule.

[0115] Continue to see Figure 3B In step 203, multiple pre-trained first models and at least one pre-trained second model are jointly trained using multiple first training samples and at least one second training sample.

[0116] Here, joint training is performed by simultaneously training multiple models (such as multiple pre-trained first models and at least one pre-trained second model), each model is learned based on a different feature set, the pre-trained first model corresponds one-to-one to the first training sample, and the pre-trained second model corresponds one-to-one to the second training sample.

[0117] In some embodiments, see Figure 3C , Figure 3C is a third flow chart of the model pruning method provided in the embodiment of the present application, for Figure 3B In step 203, multiple pre-trained first models and at least one pre-trained second model are jointly trained by using multiple first training samples and at least one second training sample. Figure 3C Steps 2031 to 2033 are implemented as described in detail below.

[0118] In step 2031, for each pre-trained first model, a first training sample corresponding to the pre-trained first model is predicted to obtain a first prediction result of the pre-trained first model.

[0119] In some embodiments, step 2031 is similar to step 102 and will not be repeated here.

[0120] In step 2032, for each pre-trained second model, a second training sample corresponding to the pre-trained second model is predicted to obtain a second prediction result of the pre-trained second model.

[0121] In some embodiments, step 2032 is similar to step 102 and will not be described again herein.

[0122] In step 2033, a first loss value is constructed based on multiple first prediction results and at least one second prediction result, and the parameters of each pre-trained first model and the parameters of each pre-trained second model are updated based on the first loss value.

[0123] Here, the first loss value is used to measure the difference between the model prediction value and the true value, and is used to guide the model to adjust parameters to improve the accuracy of the prediction.

[0124] In some embodiments, constructing a first loss value based on multiple first prediction results and at least one second prediction result in step 2033 can be achieved in the following manner: fusing multiple first prediction results and at least one second prediction result to obtain a third prediction result, and determining the second loss value based on the third prediction result; determining any two prediction results from multiple first prediction results and at least one second prediction result, performing orthogonal operations on any two prediction results to obtain orthogonal results, and determining the sum of the orthogonal results as the third loss value; adding at least one second prediction result and multiple first prediction results to obtain a fourth loss value; fusing the second loss value, the third loss value, and the fourth loss value to obtain the first loss value.

[0125] Here, the orthogonal operation is a dot product operation performed in the vector space, which is used to describe a specific relationship between two vectors.

[0126] In some embodiments, the above-mentioned “fusing multiple first prediction results and at least one second prediction result to obtain a third prediction result” can be implemented in the following way: performing weighted summation on multiple first prediction results and at least one second prediction result to obtain the third prediction result.

[0127] Continuing from the above embodiment, the above “determining the second loss value based on the third prediction result” can be implemented in the following way: determining the sample label corresponding to the first training sample; and determining the difference between the sample label and the third prediction result as the second loss value.

[0128] It should be noted that the above step of "determining the second loss value based on the third prediction result" is similar to the above step of "constructing the loss value based on the prediction results of multiple pre-trained first models", which will not be repeated here.

[0129] Continuing from the above embodiment, the above “performing an orthogonal operation on any two prediction results to obtain an orthogonal result” can be implemented in the following manner: performing a dot product on any two prediction results to obtain an orthogonal result.

[0130] Continuing from the above embodiment, the above “adding at least one second prediction result and multiple first prediction results to obtain a fourth loss value” can be implemented in the following way: constructing a prediction result set (such as [first prediction result 0.2, first prediction result 0.3, second prediction result 0.4]), the prediction result set includes at least one second prediction result and multiple first prediction results; adding multiple prediction results of the prediction result set (such as 0.9) to obtain a fourth loss value.

[0131] Continuing from the above embodiment, the above “integrating the second loss value, the third loss value, and the fourth loss value to obtain the first loss value” can be achieved in the following way: performing weighted sum (such as 0.3) of the second loss value (such as 0.2), the third loss value (such as 0.3), and the fourth loss value (such as 0.4) to obtain the first loss value.

[0132] In step 204, the first model after joint training and the second model after joint training are determined as sub-models.

[0133] It should be noted that the first model after joint training and the second model after joint training are determined as sub-models in the model to be pruned.

[0134] Through the embodiment of the present application, by jointly training multiple pre-trained first models and second models, not only the synergy between sub-models is enhanced, but also the loss value is constructed by fusing the prediction results of multiple sub-models, thereby optimizing the performance of the overall model. In the joint training process, considering the orthogonality between the prediction results of different sub-models, the generalization ability of the model is further improved through orthogonal operations, and the redundant information between models is reduced, so that each model can play a role in the final prediction task.

[0135] In step 102, the following processing is performed through the target sub-model: the input data corresponding to the target sub-model is predicted to obtain the prediction result of the target sub-model.

[0136] Here, the target sub-model is any sub-model among the multiple sub-models included in the model to be pruned. The embodiment of the present application does not limit the target sub-model. The target sub-model includes an encoder and a decoder, wherein the encoder is used to map the visualized data to a new feature space to obtain features for characterizing the latent structure of the input data, and the decoder is used to map the non-visualized data in the feature space to visualized data to obtain prediction results, so that the user can perform corresponding actions based on the prediction results.

[0137] In some embodiments, the above-mentioned "predicting the input data corresponding to the target sub-model to obtain the prediction result of the target sub-model" can be achieved in the following way: encoding the input data corresponding to the target sub-model through the encoder of the target sub-model to obtain the encoded features; decoding the encoded features through the decoder of the target sub-model to obtain the prediction result of the target sub-model.

[0138] Continuing from the above embodiment, the above step of "predicting the input data corresponding to the target sub-model to obtain the prediction result of the target sub-model" is similar to the above step of "predicting the first training sample corresponding to the pre-trained first model to obtain the prediction result of the pre-trained first model", which will not be repeated here.

[0139] In step 103, the distance between the input data corresponding to the target sub-model and other input data is determined.

[0140] Among them, other input data are input data other than the input data corresponding to the target sub-model among the multiple input data.

[0141] Here, the distance between the input data and other input data is used to characterize the correlation between the input data and other input data.

[0142] In some embodiments, the input data includes a plurality of first feature data, and the other input data includes second feature data corresponding to the first feature data one by one, see Figure 3D , Figure 3D is a fourth flow chart of the model pruning method provided in the embodiment of the present application, for Figure 3A Step 103 shown can be performed by Figure 3D Steps 1031 to 1032 are implemented as described in detail below.

[0143] In step 1031 , a candidate distance between each first feature data and the corresponding second feature data is determined.

[0144] It should be noted that the embodiment of the present application does not limit the calculation method of the candidate distance. The candidate distance can be the Euclidean distance, Manhattan distance, etc. between each first feature data and the corresponding second feature data.

[0145] In step 1032, a weighted sum is performed on multiple candidate distances to obtain the distance between the input data corresponding to the target sub-model and other input data.

[0146] For example, candidate distance A is 0.4, and candidate distance B is 0.8. When candidate distance A and candidate distance B have the same weight, that is, the weights of candidate distance A and candidate distance B are both 0.5, the distance between the input data corresponding to the target sub-model and other input data is 0.6. When candidate distance A and candidate distance B have different weights, where the weight of candidate distance A is 0.6 and the weight of candidate distance B is 0.4, the distance between the input data corresponding to the target sub-model and other input data is 0.56.

[0147] Continue to see Figure 3A In step 104, based on the prediction results of each sub-model, the target sub-model is weight predicted to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are fused to obtain the target weight of the target sub-model.

[0148] Here, the weight prediction is used to generate the prediction weight that measures the prediction results of each sub-model, and the prediction weight of the target sub-model is used to characterize the importance of the prediction results of the target sub-model.

[0149] It should be noted that the prediction weight of the target sub-model is the prediction weight of the input data corresponding to the target sub-model, and the target weight of the target sub-model is the target weight of the input data corresponding to the target sub-model.

[0150] In some embodiments, see Figure 3E , Figure 3E is a fifth flow chart of the model pruning method provided in the embodiment of the present application, for Figure 3A In step 104, based on the prediction results of each sub-model, the target sub-model is weighted predicted to obtain the prediction weight of the target sub-model, which can be obtained by Figure 3E Steps 1041A to 1042A are implemented as described in detail below.

[0151] In step 1041A, the variances of the prediction results of multiple sub-models are summed to obtain a first summation result.

[0152] In some embodiments, before step 1041A, the variance of the prediction result of each sub-model is calculated in the following manner: the mean of the prediction results of multiple sub-models is determined, and the ratio of the square of the difference between the prediction result of each sub-model and the mean to the number of sub-models is determined as the variance of the prediction result of each sub-model.

[0153] For example, given the prediction results of multiple sub-models (such as the prediction result of sub-model A is 2.3, the prediction result of sub-model B is 3, and the prediction result of sub-model C is 1.5), the mean of the prediction results of the multiple sub-models is determined (such as 2.267), and the ratio (such as 0.537) of the square of the difference between the prediction result of each sub-model (such as the prediction result of sub-model B is 3) and the mean (such as 0.733) to the number of sub-models (such as 3) is determined as the variance of the prediction result of each sub-model.

[0154] Continuing with the above example, the variances of the prediction results of multiple sub-models (such as the variance of the prediction result of sub-model A is 0.3, and the variance of the prediction result of sub-model B is 0.5) are added to obtain a first sum result (such as 0.8).

[0155] In step 1042A, the ratio of the variance of the prediction result of the target sub-model to the first summation result is determined as the prediction weight of the target sub-model.

[0156] For example, the ratio of the variance of the prediction result of the target sub-model (such as the variance of the prediction result of sub-model A is 0.3) to the first summation result (such as 0.8) is determined as the prediction weight of the target sub-model (such as 0.375).

[0157] Through the embodiments of the present application, by assigning a weight to each sub-model corresponding to its prediction stability, the method can more effectively utilize the prediction capabilities of each sub-model, thereby improving the prediction accuracy of the overall model.

[0158] In some embodiments, see Figure 3F , Figure 3F is a sixth flow chart of the model pruning method provided in the embodiment of the present application, for Figure 3A The fusion distance and prediction weight in step 104 shown in the figure are used to obtain the target weight of the target sub-model, which can be obtained by Figure 3F Steps 1041B to 1043B are implemented as described in detail below.

[0159] In step 1041B, the distances between the input data and other input data are added to obtain a second summation result, and the ratio of the preset value to the second summation result is determined as the distance weight of the target sub-model.

[0160] In some embodiments, the above step of "adding the distances between the input data and other input data respectively to obtain a second summation result" is similar to step 1041A and will not be repeated here.

[0161] For example, the distances between the input data and other input data (such as the distance between input data A and input data B is 3, and the distance between input data A and input data C is 2) are added to obtain a second sum result (such as 5), and the ratio of a preset value (such as 1) to the second sum result (such as 0.2) is determined as the distance weight of the target sub-model.

[0162] In step 1042B, the prediction weight is adjusted based on the influencing factor of the prediction weight, and the distance weight is adjusted based on the influencing factor of the distance weight.

[0163] Here, the impact factor is a numerical value used to influence a specific result (such as prediction weight or distance weight). The embodiment of the present application does not limit the impact factor. The impact factor can be a preset value or a variable value that decreases as the weight increases.

[0164] In some embodiments, the above-mentioned "adjusting the prediction weight based on the influence factor of the prediction weight" can be achieved in the following way: the product (such as 0.16) of the influence factor of the prediction weight (such as 0.2) and the prediction weight (such as 0.8) is determined as the adjusted prediction weight.

[0165] Continuing from the above embodiment, the step of "adjusting the distance weight based on the influencing factor of the distance weight" is similar to the step of "adjusting the prediction weight based on the influencing factor of the prediction weight", which will not be repeated here.

[0166] In step 1043B, the sum of the adjusted prediction weight and the adjusted distance weight is determined as the target weight of the target sub-model.

[0167] For example, the sum (eg, 0.4) of the adjusted prediction weight (eg, 0.16) and the adjusted distance weight (eg, 0.24) is determined as the target weight of the target sub-model.

[0168] Through the embodiments of the present application, by combining the prediction accuracy and the distance between the input data, the performance of the sub-model is comprehensively evaluated to ensure that the pruned model maintains good prediction results under different inputs. At the same time, the distance between the input data helps the model capture the distribution characteristics of the data, thereby improving the model's generalization ability for new data. By adjusting the prediction weights and distance weights, the importance of the sub-model is dynamically adjusted according to the model's performance on specific tasks, thereby enhancing the model's adaptability to different tasks.

[0169] Continue to see Figure 3A In step 105, based on the target weight of each sub-model, the model to be pruned is pruned to obtain a pruned model.

[0170] Among them, sub-model pruning is used to remove some sub-models in the model to be pruned.

[0171] In some embodiments, the plurality of sub-models include a plurality of first sub-models and at least one second sub-model, see Figure 3G , Figure 3G is a seventh flow chart of the model pruning method provided in the embodiment of the present application, for Figure 3A Step 105 shown can be performed by Figure 3G Steps 1051 to 1053 are implemented as described in detail below.

[0172] In step 1051, the prediction results of at least one second sub-model are added to obtain a third summed result.

[0173] Here, the first sub-model is used to predict a single feature (such as input data), and the second sub-model is used to predict the cross-features of any two features (such as any two input data). The cross-features are obtained by cross-processing any two input data.

[0174] In some embodiments, step 1051 can be implemented in the following manner: when the number of second sub-models is 1, the prediction result of the second sub-model is determined as the third sum result; when the number of second sub-models is greater than 1, the prediction results of multiple second sub-models (such as the prediction result of the second sub-model A is 0.4, and the prediction result of the second sub-model B is 0.5) are added (such as 0.9) to obtain the third sum result.

[0175] In step 1052, based on the input data corresponding to the arbitrary second sub-model and the third summation result, the first feature interaction strength of the arbitrary second sub-model is determined.

[0176] Here, the first feature interaction strength is used to characterize the degree to which the relationship between different features affects the model prediction results.

[0177] In some embodiments, step 1052 can be implemented in the following manner: based on the input data corresponding to any second sub-model, the input data corresponding to other second sub-models and the third summation result, determine the mutual information between any second sub-model and other second sub-models, wherein the other second sub-models are second sub-models other than any second sub-models in at least one second sub-model; fuse the first entropy and the second entropy to obtain the target entropy, wherein the first entropy is the entropy of the input data corresponding to any second sub-model, and the second entropy is the entropy of the input data corresponding to other second sub-models; determine the ratio between the mutual information and the target entropy as the second feature interaction strength between any second sub-model and other second sub-models; fuse the second sub-feature interaction strengths between any second sub-model and other second sub-models respectively to obtain the first feature interaction strength of any second sub-model.

[0178] Here, any second sub-model is any second sub-model in at least one second sub-model. Mutual information is used to measure the degree of mutual dependence or shared information between two random variables (such as two input data). Entropy is used to measure the uncertainty of a single random variable (such as a single input data).

[0179] In some embodiments, the above-mentioned "determining the mutual information between any second sub-model and other second sub-models based on the input data corresponding to any second sub-model, the input data corresponding to other second sub-models and the third summation result" can be achieved in the following way: determining the difference between the third entropy and the fourth entropy as the mutual information between any second sub-model and other second sub-models, wherein the third entropy is the entropy of the third summation result, and the fourth entropy is the conditional entropy of the third summation result obtained under the conditions of the input data corresponding to any second sub-model and the input data corresponding to other second sub-models.

[0180] Continuing with the above embodiment, the third entropy is obtained in the following manner: the logarithm of the third addition result (such as 0.2) (such as log(0.2)) and the product of the third addition result (such as 0.2*log(0.2)) are determined as the third entropy.

[0181] Continuing with the above embodiment, the fourth entropy is obtained in the following manner: obtaining the conditional probability of the third summation result under the conditions of the input data corresponding to any second sub-model and the input data corresponding to other second sub-models; and determining the fourth entropy by the logarithm of the conditional probability (such as 0.2) (such as log(0.2)) and the product of the conditional probability (such as 0.2*log(0.2)).

[0182] Continuing from the above embodiment, the conditional probability of obtaining the third sum result under the conditions of the input data corresponding to any second sub-model and the input data corresponding to other second sub-models is obtained in the following manner: determine the first probability (such as 0.2) that the input data corresponding to any second sub-model, the input data corresponding to other second sub-models, and the third sum result all occur; determine the second probability (such as 0.5) that the input data corresponding to any second sub-model and the input data corresponding to other second sub-models occur simultaneously; determine the ratio of the first probability to the second probability (such as 0.4) as the conditional probability of obtaining the third sum result under the conditions of the input data corresponding to any second sub-model and the input data corresponding to other second sub-models.

[0183] It should be noted that the steps of determining the first entropy and the second entropy are similar to the above-mentioned step of obtaining the third entropy, which will not be repeated here.

[0184] Following the above embodiment, the above “fusing the first entropy and the second entropy to obtain the target entropy” can be achieved in the following way: summing the first entropy and the second entropy to obtain the target entropy.

[0185] For example, given the entropy of the input data (such as 0.67) corresponding to any second sub-model, that is, the first entropy (such as 2.58), given the entropy of the input data (such as 0.5) corresponding to other second sub-models, that is, the second entropy (such as 1), the first entropy and the second entropy are added (such as 3.58) to obtain the target entropy.

[0186] Continuing from the above embodiment, the above-mentioned "fusing the second sub-feature interaction strengths between any second sub-model and other second sub-models respectively to obtain the first feature interaction strength of any second sub-model" can be achieved in the following way: performing weighted summation of multiple second sub-feature interaction strengths to obtain the first feature interaction strength of any second sub-model.

[0187] For example, a weighted sum (such as 0.3) is performed on multiple second sub-feature interaction strengths (such as the second sub-feature interaction strength between the second sub-model A and the second sub-model B is 0.2, and the second sub-feature interaction strength between the second sub-model A and the second sub-model C is 0.4) to obtain the first feature interaction strength of any second sub-model (such as the second sub-model A).

[0188] In step 1053, based on the first feature interaction strength of each second sub-model and the target weight of each first sub-model, sub-model pruning is performed on the model to be pruned to obtain a pruned model.

[0189] In some embodiments, step 1053 may perform at least one of the following processes: based on the target weight of each first sub-model, determining a first target sub-model from multiple first sub-models, and pruning the first target sub-model from the model to be pruned; based on the first feature interaction strength of each second sub-model, determining a second target sub-model from at least one second sub-model, and pruning the second target sub-model from the model to be pruned.

[0190] Continuing from the above embodiment, the above-mentioned "based on the target weight of each first sub-model, determining the first target sub-model from multiple first sub-models, and pruning the first target sub-model from the model to be pruned" can be achieved in the following way: when the target weight of the first sub-model is greater than the weight threshold, the first sub-model is determined as the first target sub-model; or, based on the target weight of each first sub-model, multiple first sub-models are sorted to obtain a sorting result, and some of the first sub-models in the sorting result are determined as the first target sub-model.

[0191] For example, when the target weight of the first submodel (such as 0.85) is greater than the weight threshold (such as 0.8), the first submodel is determined as the first target submodel, or the first submodel A corresponds to the target weight of the first submodel (such as 0.85), and the first submodel B corresponds to the target weight of the first submodel (such as 0.8), and the first submodel is sorted in ascending order based on the target weights of the first submodel (such as 0.85 and 0.8) to obtain an ascending sorting result (such as [B, A]), and the first submodel (such as A) in the latter part of the ascending sorting result is determined as the first target submodel, and the first submodel is sorted in descending order based on the target weights of the first submodel (such as 0.85 and 0.8) to obtain a descending sorting result (such as [A, B]), and the first submodel (such as A) in the former part of the descending sorting result is determined as the first target submodel.

[0192] Continuing from the above embodiment, the step of "determining a second target submodel from at least one second submodel based on the first feature interaction strength of each second submodel, and pruning the second target submodel from the model to be pruned" is similar to the step of "determining a first target submodel from multiple first submodels based on the target weight of each first submodel, and pruning the first target submodel from the model to be pruned", and will not be repeated here.

[0193] In some embodiments, after step 105, the following processing is performed: predicting the input data corresponding to each sub-model of the pruned model to obtain a fourth prediction result of each sub-model; fusing each fourth prediction result to obtain a target prediction result of the pruned model.

[0194] Through the embodiments of the present application, by calculating the feature interaction strength of the second sub-model and the target weight of the first sub-model to prune out unimportant sub-models, the computational burden of the model is reduced, making the model more suitable for running in a resource-constrained environment.

[0195] Below, an exemplary application of the model pruning method provided in an embodiment of the present application in a practical application scenario will be described.

[0196] In related technologies, traditional models combine deep learning and feature selection techniques, allowing each feature to affect the target variable through a nonlinear function. However, a large number of features leads to a slow inference rate for model predictions, and it is impossible to take into account both the interpretability of the model and the prediction accuracy of the model.

[0197] In order to solve the above problems, an embodiment of the present application proposes a model pruning method, which evaluates and prunes the model through an improved evaluation index, thereby improving the interpretability of the model and the efficiency of model reasoning without losing the prediction accuracy of the model.

[0198] Take model pruning as an example, see Figure 4 , Figure 4 is a schematic diagram of the network model structure provided in the embodiment of the present application, such as Figure 4 As shown, the network model includes an input layer 401, a main effect network 402, an interaction effect network 403 and an output layer 404. The following is an explanation of the multiple modules included in the network model provided in the embodiment of the present application.

[0199] 1) Input layer 401

[0200] Here, the input layer 401 is used to input different input data into the main effect network 402 and the interaction effect network 403 (both the main effect network 402 and the interaction effect network 403 are sub-models in the model to be pruned), and the input data is the data after data preprocessing of the initial data. The specific process of data preprocessing can be achieved in the following way: the missing values ​​of the initial data are interpolated using the mean or mode of the data, wherein the mean of the data is used to characterize the numerical characteristics of the data, and the mode of the data is used to characterize the categorical characteristics of the data. Then, the interpolated data is standardized, and the categorical variable encoding (such as one-hot encoding) is performed on the standardized data to obtain the input data. The initial data may be user attributes (such as user name, gender, age, etc.), product information (such as product name, attributes, price, etc.), specific content (such as the time when the user receives the product push information, the time when the user clicks on the product push information, etc.) and product number (such as the identifier of the product). The input data input to the main effect network 402 may be the coded features of the user's attributes after data preprocessing, the coded features of the product information after data preprocessing, the coded features of the specific content after data preprocessing and the coded features of the product number after data preprocessing. The input data input to the interactive effect network 403 may be the cross features of any two coded features in the input data input to the main effect network 402 (such as cross feature 1 and cross feature 2). The cross features of any two coded features may be the concatenation or weighted sum of any two features.

[0201] 2) Main effect network 402 (i.e., the first sub-model)

[0202] Here, the main effect network is a network composed of single variable input sub-networks 4021, and the single variable input sub-network 4021 is a sub-network with a single variable input.

[0203] 3) Interaction effect network 403 (i.e., the second sub-model)

[0204] Here, the interaction effect network 403 is a network composed of a two-variable input cross-interaction network 4031, wherein the two-variable input cross-interaction network is a network composed of two-variable inputs with feature crossover.

[0205] Since each dimension of features is only related to the change value of the feature itself, the impact of each dimension of features on the final result can be clearly calculated from the network model. The performance of the model is guaranteed through the neural network (such as the single variable input sub-network 4021 and the two-variable input cross-interaction network 4031), while the interactive operations are simplified and isolated.

[0206] See also Figure 5 , Figure 5 is a schematic diagram of the sub-network model structure provided in the embodiment of the present application, such as Figure 5 As shown, the input vector is input into the encoder 501 of the sub-network (such as the univariate input sub-network 4021 and the bivariate input cross-interaction network 4031) for encoding to obtain the encoded vector, and the encoded vector is input into the decoder 502 of the sub-network for decoding to obtain the output vector (i.e., the prediction result).

[0207] 4) Output layer 404

[0208] Here, the output variables of each dimension of features are added together to obtain the final total prediction result, which is used to calculate the prediction loss of the model; the output variables of each dimension of features are fused to obtain the sparsity loss of the network model; based on any two output variables, the orthogonal loss of the model can be calculated; the prediction loss, sparsity loss and orthogonal loss of the model are used to construct a loss function to update the parameters of the model and obtain the trained network model.

[0209] The process of training the network model through the above modules is described in detail below.

[0210] The process of training the network model is carried out in two stages. The two stages of the training model provided in the embodiment of the present application are explained below.

[0211] In the first stage, the main effect network 402 is pre-trained.

[0212] Here, prediction loss is used for training, and prediction loss is the difference between the data label corresponding to the input data and the total prediction result.

[0213] In the second stage, the interaction effect network 403 is introduced.

[0214] Here, a loss function is used for training, and the process of obtaining the loss function is shown in Formula 1.

[0215] L=L pred +λ1*L ortho +λ2*L sparsity (1)

[0216] Among them, the loss function L (i.e., the first loss value) includes the prediction loss L pred (i.e., the second loss value), orthogonal loss L ortho (i.e. the third loss value) and the sparsity loss L sparsity (i.e., the fourth loss value), λ1 and λ2 are trade-off parameters used to control the influence of orthogonal loss and sparsity loss.

[0217] The loss function ensures prediction accuracy through prediction loss, learns independent and non-redundant feature representations through orthogonal loss, realizes automatic feature selection through sparsity loss, and balances different objectives by adjusting λ1 and λ2 to improve the interpretability and generalization ability of the model.

[0218] Prediction loss is the most basic loss function, which is used to measure the difference between the model prediction value and the actual value. The calculation process of prediction loss is shown in Formula 2.

[0219] L pred =E[L(Y,f(X))] (2)

[0220] Where Y is the actual target value, i.e., the real data label of input X, f(X) is the model’s prediction of input X, and L is the selected loss function (such as mean square error or cross entropy), which improves the model’s prediction accuracy by minimizing the prediction error.

[0221] Orthogonal loss is used to reduce the redundancy and correlation between features, prompting the model to learn more independent and orthogonal feature representations. The calculation process of orthogonal loss is shown in Formula 3.

[0222]

[0223] in, and are the learned representations of different features, that is, different output vectors (i.e., prediction results). <·,·> represents the inner product operation to minimize the correlation between different feature representations. By penalizing non-orthogonal feature representations, the independence between features is ensured, thereby reducing feature redundancy and improving the interpretability and generalization ability of the model.

[0224] Sparsity loss is used to encourage the model to learn sparse feature representations, that is, the weights of most features are close to zero. The calculation process of sparsity loss is shown in Formula 4.

[0225]

[0226] in, It is feature X i The corresponding output vector is,

[0227] Sparsity loss forces the weights of most features to approach zero and retains only the most important features. Sparsity is achieved by introducing regularization, which helps select features, reduce overfitting, and improve the interpretability and computational efficiency of the model.

[0228] During the model training process, by optimizing this loss function, high prediction accuracy, feature independence, and model sparsity can be achieved simultaneously, providing accurate and explainable prediction results in complex scenarios such as finance.

[0229] After obtaining the trained network model, model pruning is performed using the trained network model, and an improved Sobol index of the input data and the feature interaction strength of the input data are calculated according to the input data, and the network model is pruned according to the improved Sobol index of the input data and the feature interaction strength of the input data, wherein the calculation process of the improved Sobol index is shown in Formula 5, and the calculation process of the feature interaction strength is shown in Formula 6.

[0230]

[0231]

[0232] Among them, S i It is feature X i Sobol index (i.e., prediction weight), α is the feature X i The coefficient of the Sobol index (i.e. the influencing factor of the prediction weight), W(X i ,X j ) is feature X i (i.e., the input data corresponding to any second sub-model) and X j (i.e., the input data corresponding to the other second sub-model) distribution distance (such as Wasserstein distance), Wasserstein distance is used to measure the difference between two probability distributions, X i and X j are different input data, is the reciprocal of the sum of the distribution distances (i.e., the distance weight), β is the coefficient of the reciprocal of the sum of the distribution distances (i.e., the influencing factor of the distance weight), I(X i ,X j ; Y) is the joint distribution (X i ,X j ) and the target variable Y (i.e., the third summation result), H(X i ) is feature X i The entropy (i.e., the first entropy), IS(X i ,X j ) is the joint distribution (X i ,X j ) (i.e., the second feature interaction strength).

[0233] The improved Sobol coefficient further enhances the ability to evaluate feature importance by combining the Wasserstein distance, especially in financial trading scenarios with high dimensions and possible redundant features. This improvement enables the model to more accurately identify truly important features, improving the model's interpretability and predictive performance.

[0234] The calculation process of the Sobol index is described in detail below. Given a network model Y = f(X i ,X j ,...,X k ), where Y is the output variable (i.e., the predicted result), X i ,X j ,...,X k are k input variables (i.e., input data). The total variance V(Y) (i.e., the first summation result) output by the model is decomposed as shown in Formula 7.

[0235] V(Y)=∑ i V i +∑ i ∑ j>i V ij +…+V i1,2,…,k (7)

[0236] Among them, V i is a single variable X i The variance caused by (i.e., the variance of the prediction results of the target sub-model), V ij is the variable X i and X j The variance caused by the interaction of i1,2,…,k It is the variance caused by the interaction of all variables. The calculation process of the first-order Sobol coefficient (i.e., prediction weight) is shown in Formula 8.

[0237] S i =V i / V(Y) (8)

[0238] Among them, S i Represents variable X i The proportion of the individual contribution to the output variance.

[0239] By calculating the Sobol coefficient, the entire input space is taken into account, not just local changes. It is suitable for various types of models, including nonlinear and non-monotonic models. At the same time, the interaction between input variables can be quantified, and precise numerical values ​​are provided to compare the importance of different input variables. In addition, the sum of all first-order Sobol coefficients is less than or equal to 1. When the sum of the first-order Sobol coefficients is equal to 1, it indicates that the network model is completely additive.

[0240] Based on the calculated improved Sobol index and feature interaction strength, unimportant features and the neural networks corresponding to the features are pruned regularly, and the hyperparameters λ1, λ2 and the network structure are dynamically adjusted using Bayesian optimization.

[0241] See also Fig. 6A , Fig. 6AThis is a first data processing flow chart provided in an embodiment of the present application. The first process of processing data provided in an embodiment of the present application is explained below.

[0242] In step 601, data is input.

[0243] Here, the original data set is loaded from the data source (such as a database, CSV file, etc.), and the data structure of the data in the original data set is preliminarily checked to confirm the feature type and target variable of the data.

[0244] In step 602, data cleaning is performed on the input data.

[0245] Here, the input data is cleaned through at least one of step 6021 and step 6022.

[0246] In step 6021, the missing values ​​of the input data are processed.

[0247] For numerical features, the mean or median is used to fill missing values ​​of the input data. For categorical features, the mode is used to fill missing values ​​of the input data or create a new "unknown" category.

[0248] In step 6022, outliers in the input data are processed.

[0249] Here, firstly, a box plot or a standard score method is used to identify outliers in the input data, and then the outliers in the input data are processed in at least one of the following ways: deletion, replacement, and retention.

[0250] In step 603, feature processing is performed on the cleaned data.

[0251] Here, feature processing is performed on the cleaned data through at least one of steps 6031 to 6033.

[0252] In step 6031, feature scaling is performed on the cleaned data.

[0253] Here, the cleaned data belonging to the numerical features is standardized by standard scores or normalized by linear functions (Min-Max Scaling).

[0254] In step 6032, feature encoding is performed on the cleaned data.

[0255] Here, the cleaned data belonging to the categorical features are one-hot encoded or label encoded.

[0256] In step 6033, features are created.

[0257] Here, features are created in at least one of the following ways: creating new features based on domain knowledge; or, combining existing features; or, interactively processing existing features to generate interactive features.

[0258] See also Figure 6B , Figure 6B is a second data processing flow chart provided in an embodiment of the present application, Figure 6B Through node A and Fig. 6A The second process of processing data provided in the embodiment of the present application is explained below.

[0259] In step 604, model training is performed on the network model.

[0260] Here, the model training includes a training process of multiple stages. The multiple stages of model training are described in detail below. Among them, the model training process of the first stage can be implemented through step 6041, and the model training process of the second stage can be implemented through step 6042.

[0261] In step 6041, the main effect network (ie, the pre-trained first model) is pre-trained.

[0262] Here, only a single variable (i.e., the first training sample) is used as input into the sub-network of the main effect network, and the prediction loss function is optimized according to the output prediction result to obtain the pre-trained main effect network (i.e., the first model after pre-training). The prediction loss function is the difference between the output prediction result and the label result corresponding to the single variable.

[0263] In step 6042, the interaction effect network (ie, the pre-trained second model) is introduced.

[0264] Here, a dual variable (i.e., the second training sample) is added and input into a sub-network of the cross-interaction network, and the complete loss function is optimized according to the output prediction result, wherein the complete loss function includes prediction loss, orthogonal loss, and sparsity loss.

[0265] In step 605, feature importance is evaluated for each feature input into the network model.

[0266] Here, step 6051 is performed to obtain an improved evaluation index, and the importance of the feature is evaluated according to the improved evaluation index.

[0267] In step 6051, an improved evaluation index of the feature (such as an improved Sobol index) is calculated.

[0268] Here, the traditional Sobol index and the Wasserstein distance between features are calculated, and the improved Sobol index is calculated by combining the Sobol index and the Wasserstein distance.

[0269] See also Figure 6C , Figure 6C is a third data processing flow chart provided in an embodiment of the present application, Figure 6C Through Node B and Figure 6B The third process of processing data provided in the embodiment of the present application is explained below.

[0270] In step 606, feature interaction analysis is performed on each feature of the input network model.

[0271] Here, step 6061 is executed to obtain the feature interaction strength of the features, and the interaction between the features is analyzed based on the feature interaction strength.

[0272] In step 6061, the feature interaction strength of each feature of the input network model is calculated.

[0273] Here, the following processing is performed on any feature pair: the mutual information between the feature pair is calculated, and the entropy of a single feature is calculated, and then the feature interaction strength is calculated based on the mutual information between the feature pair and the entropy of each feature in the feature pair.

[0274] In step 607, model pruning is performed on the model to be pruned.

[0275] Here, step 607 can be implemented through step 6071.

[0276] In step 6071, the model to be pruned is pruned based on the importance index.

[0277] Here, an importance threshold is set, and based on the improved Sobol index (i.e., target weight) and the importance threshold corresponding to the improved Sobol index, the sub-networks in the unimportant main effect network (i.e., the first target sub-model) and the features corresponding to the sub-networks are pruned; based on the feature interaction strength and the importance threshold corresponding to the feature interaction strength, the sub-networks of unimportant interaction effects (i.e., the second target sub-model) and the features corresponding to the sub-networks are pruned.

[0278] In step 608, the pruned model is evaluated.

[0279] Here, the model is evaluated by executing the performance indicators obtained in step 6081, evaluating the model performance, and cross-validation is performed to ensure the stability of the model.

[0280] In step 6081, the performance index of the pruned model is calculated.

[0281] Here, the performance indicator of the pruned model can be the AUC indicator.

[0282] In step 609, the prediction results output by the pruned model are interpreted.

[0283] Here, the prediction results output by the model are explained through at least one of step 6091 and step 6092.

[0284] In step 6091, a feature importance visualization is generated.

[0285] Here, the modified Sobol index is visualized to explain the importance of individual features.

[0286] In step 6092, a feature interaction heat map is generated.

[0287] Here, the feature interaction intensity heat map is visualized to explain the interaction relationship between features. The partial dependency plot (PDP) or Shapley additive explanation (SHAP) value can also be generated to explain the impact of features on the prediction results.

[0288] In a given pension user prediction scenario, the model evaluation method provided in this application ranks the importance of the given key features as follows: 1) the date of holding stable assets; 2) user age; 3) the number of funds held; 4) the number of visits in the last 30 days.

[0289] The obtained clear feature importance ranking provides direct guidance for business decision-making in the pension user prediction scenario.

[0290] In a given short-term data package recommendation scenario, the model pruning method provided by this application reduces the number of features and the dimension of the features without sacrificing performance. At the same time, it reduces the feature storage space and model training time, and improves the model's reasoning speed. The network model provided by this application generates a relationship diagram between the number of virtual props held and the conversion rate, showing the positive correlation between the two. Figure 7 , Figure 7 is a schematic diagram of the interaction interface between a user and a product provided in an embodiment of the present application, Figure 7 The interactive interface 701 includes historical information related to the product 7011, the name of the product 7012, relevant information about the product (such as the annualized rate of return of the product) 7013, and a button 7014 for the user to interact with the product. In response to the click operation of the button 7014, a certain number of products are placed in the user's backpack, that is, the user executes the operation of purchasing the product.

[0291] After the user interacts with the product, the user-related information (such as age, gender, etc.) and product-related information (such as type, revenue, use, etc.) are counted, and the feature importance of each relevant information is determined according to the model evaluation method provided in the embodiment of the present application, see Figure 8 , Figure 8 is a schematic diagram of the first feature importance provided in the embodiment of the present application, such as Figure 8 As shown, the first important feature is the date of stable asset holding 801, the second most important is the user's age 802, followed by the number of products held 803 and the number of visits in the past 30 days 804.

[0292] Statistics of different ages and the predicted purchase conversion rates corresponding to each age, see Fig. 9A , Fig. 9A This is the first schematic diagram of the relationship between age and purchase conversion rate provided by the embodiment of the present application. Fig. 9A As shown in , as the number of products held increases, when the age is greater than the age threshold, the purchase conversion rate 901 predicted by the main effect network increases linearly with the increase of age. Fig. 9B , Fig. 9B This is the first schematic diagram of the distribution density of age provided in the embodiment of the present application. Fig. 9B As shown, in the input data provided by the embodiment of the present application, data 902 of users aged 30 to 40 accounts for nearly 40% of the data of users of all age groups, that is, nearly 40% of the input data provided by the embodiment of the present application is data of users aged 30 to 40.

[0293] The relationship between the user’s age at the time of purchase and the product’s account allocation ratio is shown in Table 1.

[0294] Table 1 Distribution ratio relationship table

[0295] Age at time of purchase Account allocation ratio of the product 18-30 years old 5％ 31-40 years old 10％ 41-50 years old 20％ 51-60 years old 40％ Over 60 years old 100％

[0296] The account allocation ratio is used to represent the percentage of the user's expenditure on purchasing products to the user's total expenditure.

[0297] See also Fig. 9C , Fig. 9C Schematic diagram of the relationship between the number of products held and the purchase conversion rate provided in the embodiment of the present application. Fig. 9C As shown in Figure 2, as the number of products held increases, the purchase conversion rate predicted by the main effect network increases by 903, the marginal benefits of the first k funds (such as 4) are the highest, and the slope of the growth is fixed afterwards. Fig.9D , Fig.9D Schematic diagram of the distribution density of the number of products held provided in the embodiment of the present application. Fig.9DAs shown, in the input data provided by the embodiment of the present application, users who hold a number of products less than a quantity threshold (such as 17) account for a high proportion (such as greater than 95%) of the total users, while users who hold a number of products greater than or equal to the quantity threshold (such as 19) account for a low proportion (such as less than 5%) of the total users.

[0298] In a given traffic data package recommendation scenario, the model provided in this application is used to predict the possibility of users purchasing traffic data packages and recommend package types. Fig.10 , Fig.10 is a schematic diagram of the interaction interface between a user and a traffic data package provided in an embodiment of the present application, Fig.10 The interactive interface 1001 includes a recommendation module 1002 and an activity area module 1003. The recommendation module 1002 includes a recommended traffic data package A 10021 and a purchase button 10022 corresponding to the traffic data package A 10021. The activity area module 1003 includes a recommended traffic data package B 10031 and a traffic data package C 10032. In response to the query operation of the traffic data package B, it jumps to a new interface for introducing the traffic data package B.

[0299] After the user interacts with the product, the relationship between the duration of traffic data package distribution and the purchase conversion rate is counted, see Fig.11 , Fig.11 is a schematic diagram of the importance of the second feature provided in the embodiment of the present application, such as Fig.11 As shown, the first important feature is the click-through rate 1101 of the package with a traffic duration of 1 day, the second most important is the click-through rate 1102 of the package with a traffic duration of 7 days, and the next most important is the click-through rate 1103 of the package with a traffic duration of 30 days. The click-through rate of the package with a traffic duration of 1 day is more important than the click-through rate of the package with a traffic duration of 7 days. The characteristics of this scenario are mainly dominated by the attributes of the items, and the user's attributes account for a relatively low proportion.

[0300] The relationship between the conversion rate and purchase conversion rate corresponding to different days is counted. The conversion rate refers to the ratio of "conversion" completed in traffic ("conversion" here can be the sale of physical goods, the consumption of online services, the submission of information, etc. The daily conversion rate is the ratio of conversion completed within one day. Take the package with a 7-day traffic distribution period as an example, see Fig. 12A , Fig. 12A Schematic diagram of the relationship between the one-day conversion rate and the purchase conversion rate provided in the embodiment of the present application. Fig. 12A As shown, as the one-day conversion rate increases, the purchase conversion rate 1201 predicted by the main effect network increases with the one-day conversion rate. Fig. 12B , Fig. 12BSchematic diagram of the distribution density of the one-day conversion rate provided in the embodiment of the present application. Fig. 12B As shown, in the input data provided by the embodiment of the present application, the one-day conversion rate 1202 less than the conversion rate threshold (such as 0.02) accounts for nearly 80% of all one-day conversion rates.

[0301] Statistics on the relationship between different click-through conversion rates and purchase conversion rates. Click-through conversion rate is the proportion of conversions after "viewing" or "clicking" an ad, and only counts users who "view" or "click". Fig. 12C , Fig. 12C Schematic diagram of the relationship between click conversion rate and purchase conversion rate provided in the embodiment of the present application. Fig. 12C As shown in , as the click-through conversion rate increases, the purchase conversion rate 1203 predicted by the main effect network increases with the click-through conversion rate. Fig.12D , Fig.12D Schematic diagram of the distribution density of click-through conversion rate provided by the embodiment of the present application. Fig.12D As shown, in the input data provided by the embodiment of the present application, the click-through conversion rate 1204 less than the click-through conversion rate threshold (such as 0.02) accounts for nearly 60% of all conversion rates. The higher the click-through conversion rate, the higher the purchase, but the impact is not as high as the conversion rate of one day.

[0302] To analyze the relationship between different user ages and purchase conversion rates, see Fig.12E , Fig.12E This is a second schematic diagram of the relationship between age and purchase conversion rate provided in the embodiment of the present application. Fig.12E As shown, when the age is less than the age threshold, as the user age increases, the purchase conversion rate 1205 predicted by the main effect network increases with age. When the user age is greater than the age threshold, as the user age increases, the purchase conversion rate 1205 predicted by the main effect network decreases with age. The users who are most likely to convert are concentrated between 20 and 40 years old. Fig.12F 2 is a second schematic diagram of the distribution density of ages provided in the embodiment of the present application. Fig.12F ,like Fig.12F As shown, among the input data provided by the embodiment of the present application, data 1206 of users under the age of 10 accounts for nearly 40% of the data of users of all age groups. It can be seen that the timeliness of the traffic recommendation scenario is very important. The model mainly recommends items that are selling well in the recent period, which requires strong timeliness capture, such as the real-time nature of data and recommendation models.

[0303] In summary, the embodiments of the present application improve the accuracy and robustness of feature importance evaluation by proposing an improved Sobol index combined with the Wasserstein distance, quantify the feature interaction strength by the ratio of mutual information and entropy, effectively capture complex nonlinear relationships, and at the same time, separate the training of main effects and interaction effects to improve model performance and interpretability, and reduce feature collinearity by calculating the orthogonal loss function, thereby improving the stability and interpretability of the model. Finally, model pruning is performed to achieve adaptive simplification of the model based on the improved Sobol index and feature interaction strength.

[0304] The following is a description of an exemplary structure of the model pruning device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the model pruning device 555 of the memory 550 may include:

[0305] The data acquisition module 5551 is used to obtain the model to be pruned and multiple input data, wherein the model to be pruned includes multiple sub-models, and the input data corresponds to the sub-models one by one.

[0306] The prediction module 5552 is used to perform the following processing through the target sub-model: predict the input data corresponding to the target sub-model to obtain the prediction result of the target sub-model.

[0307] The pruning module 5553 is used to determine the distance between the input data corresponding to the target sub-model and other input data, wherein the other input data is input data other than the input data corresponding to the target sub-model among multiple input data; based on the prediction result of each sub-model, weight prediction is performed on the target sub-model to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are integrated to obtain the target weight of the target sub-model; based on the target weight of each sub-model, sub-model pruning is performed on the model to be pruned to obtain the pruned model.

[0308] In some embodiments, the pruning module 5553 is also used to add the prediction results of at least one second sub-model to obtain a third summed result; determine the first feature interaction strength of any second sub-model based on the input data corresponding to any second sub-model and the third summed result; perform sub-model pruning on the pruned model based on the first feature interaction strength of each second sub-model and the target weight of each first sub-model to obtain a pruned model, wherein the multiple sub-models include multiple first sub-models and at least one second sub-model.

[0309] In some embodiments, the pruning module 5553 is also used to determine the mutual information between any second submodel and other second submodels based on the input data corresponding to any second submodel, the input data corresponding to other second submodels and the third summation result, wherein the other second submodels are second submodels other than any second submodel in at least one second submodel; fuse the first entropy and the second entropy to obtain the target entropy, wherein the first entropy is the entropy of the input data corresponding to any second submodel, and the second entropy is the entropy of the input data corresponding to other second submodels; determine the ratio between the mutual information and the target entropy as the second feature interaction strength between any second submodel and other second submodels; fuse the second sub-feature interaction strengths between any second submodel and other second submodels respectively to obtain the first feature interaction strength of any second submodel.

[0310] In some embodiments, the pruning module 5553 is also used to perform at least one of the following processes: based on the target weight of each first sub-model, determining a first target sub-model from multiple first sub-models, and pruning the first target sub-model from the model to be pruned; based on the first feature interaction strength of each second sub-model, determining a second target sub-model from at least one second sub-model, and pruning the second target sub-model from the model to be pruned.

[0311] In some embodiments, the training module 5554 is used to obtain multiple pre-trained first models and at least one pre-trained second model, and to obtain multiple first training samples and at least one second training sample, wherein the first training samples correspond one-to-one to the pre-trained first models, the second training samples correspond one-to-one to the pre-trained second models, and the second training samples are obtained by performing feature cross-processing on any two first training samples; for each pre-trained first model, the pre-trained first model is pre-trained through the corresponding first training sample to obtain the pre-trained first model; through multiple first training samples and at least one second training sample, multiple pre-trained first models and at least one pre-trained second model are jointly trained; the jointly trained first model and the jointly trained second model are determined as sub-models.

[0312] In some embodiments, the training module 5554 is also used to predict the first training sample corresponding to the pretrained first model for each pretrained first model to obtain a first prediction result of the pretrained first model; predict the second training sample corresponding to the pretrained second model for each pretrained second model to obtain a second prediction result of the pretrained second model; construct a first loss value based on multiple first prediction results and at least one second prediction result, and update the parameters of each pretrained first model and the parameters of each pretrained second model based on the first loss value.

[0313] In some embodiments, the training module 5554 is also used to fuse multiple first prediction results and at least one second prediction result to obtain a third prediction result, and determine a second loss value based on the third prediction result; determine any two prediction results from multiple first prediction results and at least one second prediction result, and perform orthogonal operations on any two prediction results to obtain orthogonal results, and determine the sum of the orthogonal results as the third loss value; add at least one second prediction result and multiple first prediction results to obtain a fourth loss value; and fuse the second loss value, the third loss value, and the fourth loss value to obtain the first loss value.

[0314] In some embodiments, the training module 5554 is also used to perform one of the following processes: obtaining multiple first initial samples, standardizing each first initial sample to obtain a standardized sample, and encoding the standardized sample to obtain a first training sample; obtaining multiple second initial samples, classifying each second initial sample to obtain a category of each second initial sample, and determining the second initial samples belonging to the same category as the first training sample.

[0315] In some embodiments, the pruning module 5553 is also used to add the distances between the input data and other input data respectively to obtain a second summation result, and determine the ratio of the preset value to the second summation result as the distance weight of the target sub-model; based on the influencing factor of the prediction weight, adjust the prediction weight, and based on the influencing factor of the distance weight, adjust the distance weight; and determine the sum of the adjusted prediction weight and the adjusted distance weight as the target weight of the target sub-model.

[0316] In some embodiments, the pruning module 5553 is also used to determine the candidate distance between each first feature data and the corresponding second feature data; perform weighted summation on multiple candidate distances to obtain the distance between the input data corresponding to the target sub-model and other input data, wherein the input data includes multiple first feature data, and the other input data includes second feature data corresponding one-to-one to the first feature data.

[0317] In some embodiments, the pruning module 5553 is also used to add the variances of the prediction results of multiple sub-models to obtain a first summed result; and determine the ratio of the variance of the prediction result of the target sub-model to the first summed result as the prediction weight of the target sub-model.

[0318] An embodiment of the present application provides a computer program product, which includes computer executable instructions. The computer executable instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the model pruning method described above in the embodiment of the present application.

[0319] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the model pruning method provided by the present application embodiment, for example, Figures 3A to 3G The model pruning method shown.

[0320] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0321] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0322] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0323] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0324] To summarize, the input data corresponding to the target sub-model is predicted to obtain the prediction result of the target sub-model, the distance between the input data corresponding to the target sub-model and other input data is determined, and based on the prediction result of each sub-model, the target sub-model is weighted predicted to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are fused to obtain the target weight of the target sub-model. Based on the target weight of each sub-model, the model to be pruned is sub-model pruned to obtain the pruned model. In this way, the distance between the input data of the model and the prediction ability of the model are comprehensively considered when pruning the sub-model, which improves the accuracy of the sub-model pruning, thereby ensuring the performance of the pruned model.

[0325] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model pruning method, characterized in that: The method comprises: Acquire a model to be pruned and a plurality of input data, wherein the model to be pruned includes a plurality of sub-models, and the input data corresponds to the sub-models one by one; The following processing is performed by the target sub-model: predicting the input data corresponding to the target sub-model to obtain a prediction result of the target sub-model; Determine a distance between the input data corresponding to the target sub-model and other input data, wherein the other input data is the input data other than the input data corresponding to the target sub-model among the multiple input data; Based on the prediction results of each of the sub-models, weight prediction is performed on the target sub-model to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are fused to obtain the target weight of the target sub-model; Based on the target weight of each sub-model, sub-model pruning is performed on the model to be pruned to obtain a pruned model.

2. The method according to claim 1, characterized in that The multiple sub-models include multiple first sub-models and at least one second sub-model, and the sub-model pruning of the model to be pruned based on the target weight of each of the sub-models to obtain a pruned model includes: Adding the prediction results of the at least one second sub-model to obtain a third summed result; Determine a first feature interaction strength of any second sub-model based on the input data corresponding to any second sub-model and the third sum result; Based on the first feature interaction strength of each of the second sub-models and the target weight of each of the first sub-models, sub-model pruning is performed on the model to be pruned to obtain the pruned model.

3. The method according to claim 2, characterized in that The determining the first feature interaction strength of any second sub-model based on the input data corresponding to any second sub-model and the third addition result includes: Determine the mutual information between any of the second sub-models and the other second sub-models based on the input data corresponding to any of the second sub-models, the input data corresponding to the other second sub-models, and the third addition result, wherein the other second sub-models are the second sub-models other than any of the second sub-models in the at least one second sub-model; The first entropy and the second entropy are combined to obtain a target entropy, wherein the first entropy is the entropy of the input data corresponding to any of the second sub-models, and the second entropy is the entropy of the input data corresponding to the other second sub-models; Determine the ratio between the mutual information and the target entropy as the second feature interaction strength between any of the second sub-models and the other second sub-models; The second sub-feature interaction strengths between any of the second sub-models and the other second sub-models are fused to obtain the first feature interaction strength of any of the second sub-models.

4. The method according to claim 2, characterized in that: The performing sub-model pruning on the model to be pruned based on the first feature interaction strength of each of the second sub-models and the target weight of each of the first sub-models to obtain the pruned model includes: Perform at least one of the following: Based on the target weight of each of the first sub-models, determine a first target sub-model from the multiple first sub-models, and prune the first target sub-model from the model to be pruned; Based on the first feature interaction strength of each of the second sub-models, a second target sub-model is determined from the at least one second sub-model, and the second target sub-model is pruned from the model to be pruned.

5. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the model to be pruned, the method further includes: Acquire multiple pre-trained first models and at least one pre-trained second model, and acquire multiple first training samples and at least one second training sample, wherein the first training samples correspond one-to-one to the pre-trained first models, the second training samples correspond one-to-one to the pre-trained second models, and the second training samples are obtained by performing feature cross processing on any two of the first training samples; For each of the pre-trained first models, pre-train the pre-trained first model using the corresponding first training sample to obtain a pre-trained first model; Jointly training the plurality of pre-trained first models and the at least one pre-trained second model using the plurality of first training samples and the at least one second training sample; The first model after joint training and the second model after joint training are determined as the sub-models.

6. The method according to claim 5, characterized in that The step of jointly training the plurality of pre-trained first models and the at least one pre-trained second model by using the plurality of first training samples and the at least one second training sample comprises: For each of the pre-trained first models, predict the first training samples corresponding to the pre-trained first model to obtain a first prediction result of the pre-trained first model; For each of the pre-trained second models, predicting the second training samples corresponding to the pre-trained second model to obtain a second prediction result of the pre-trained second model; A first loss value is constructed based on multiple first prediction results and at least one second prediction result, and parameters of each pre-trained first model and parameters of each pre-trained second model are updated based on the first loss value.

7. The method according to claim 6, characterized in that The constructing a first loss value based on the plurality of first prediction results and at least one second prediction result comprises: fusing a plurality of the first prediction results and at least one of the second prediction results to obtain a third prediction result, and determining a second loss value based on the third prediction result; Determine any two prediction results from a plurality of the first prediction results and at least one of the second prediction results, perform an orthogonal operation on the any two prediction results to obtain an orthogonal result, and determine the sum of the orthogonal results as a third loss value; Adding at least one of the second prediction results and a plurality of the first prediction results to obtain a fourth loss value; The second loss value, the third loss value, and the fourth loss value are integrated to obtain the first loss value.

8. The method according to claim 5, characterized in that The obtaining of a plurality of first training samples comprises: Perform one of the following actions: Acquire multiple first initial samples, perform standardization processing on each of the first initial samples to obtain a standardized sample, and encode the standardized sample to obtain the first training sample; A plurality of second initial samples are obtained, each of the second initial samples is classified to obtain a category of each of the second initial samples, and the second initial samples belonging to the same category are determined as the first training samples.

9. The method according to any one of claims 1 to 4, characterized in that: The fusing the distance and the prediction weight to obtain the target weight of the target sub-model includes: Adding the distances between the input data and the other input data respectively to obtain a second summation result, and determining a ratio of a preset value to the second summation result as a distance weight of the target submodel; Based on the influencing factor of the prediction weight, adjusting the prediction weight, and based on the influencing factor of the distance weight, adjusting the distance weight; The sum of the adjusted prediction weight and the adjusted distance weight is determined as the target weight of the target sub-model.

10. The method according to any one of claims 1 to 4, characterized in that: The input data includes a plurality of first feature data, and the other input data includes second feature data corresponding to the first feature data one by one; The determining the distance between the input data corresponding to the target sub-model and other input data includes: determining a candidate distance between each of the first feature data and the corresponding second feature data; A weighted sum is performed on the multiple candidate distances to obtain the distance between the input data corresponding to the target sub-model and other input data.

11. The method according to any one of claims 1 to 4, characterized in that: The step of performing weight prediction on the target sub-model based on the prediction result of each sub-model to obtain the prediction weight of the target sub-model includes: Adding the variances of the prediction results of the plurality of sub-models to obtain a first summed result; The ratio of the variance of the prediction result of the target sub-model to the first summation result is determined as the prediction weight of the target sub-model.

12. A model pruning device, characterized in that: The device comprises: A data acquisition module, used to obtain a model to be pruned and a plurality of input data, wherein the model to be pruned includes a plurality of sub-models, and the input data corresponds to the sub-models one by one; A prediction module is used to perform the following processing through the target sub-model: predict the input data corresponding to the target sub-model to obtain the prediction result of the target sub-model; A pruning module is used to determine the distance between the input data corresponding to the target sub-model and other input data, wherein the other input data is the input data other than the input data corresponding to the target sub-model among the multiple input data; based on the prediction result of each of the sub-models, weight prediction is performed on the target sub-model to obtain the prediction weight of the target sub-model, and the distance and the prediction weight are integrated to obtain the target weight of the target sub-model; based on the target weight of each of the sub-models, sub-model pruning is performed on the model to be pruned to obtain a pruned model.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the model pruning method according to any one of claims 1 to 11 when executing computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the model pruning method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the model pruning method according to any one of claims 1 to 11 is implemented.

Images