Model storage method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a model storage method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: acquiring a full modal output model; in response to the received model acquisition information, determining an input mode set and business output information; for each at least one input mode, executing a first generation step: carrying out model cutting on the full-quantity mode output model to generate a cut multi-mode output model; obtaining a multi-modal data set; performing multi-modal data preprocessing on the multi-modal data set to generate a preprocessed data set; performing model training on the cut multi-modal output model to generate a trained multi-modal output model; and carrying out model storage and model deployment on the obtained at least one trained multi-modal output model. According to the embodiment, model landing and storage for the target business scene can be accurately and efficiently realized, so that model use in the subsequent target business scene is facilitated.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a model storage method, device, electronic device, and computer-readable medium. Background Art

[0002] At present, with the continuous development of society, various intelligent applications have also appeared widely in people's daily lives. For model generation under predetermined business scenarios, the usual method is: first, obtain a training data set. Then, design a model structure for the predetermined business scenario to obtain an initial business scenario model. Finally, according to the training data set, the above initial business scenario model is trained to generate a business scenario model that meets the agreed business scenario.

[0003] However, when the above method is used to generate models for predetermined business scenarios, the following technical problems often occur:

[0004] For scheduled business scenarios with massive demand, targeted model structure design is required, which will waste a lot of model design resources, resulting in a long implementation time of the business scenario model and inconvenience in using the model in the business scenario.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a model storage method, an apparatus, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a model storage method, comprising: obtaining a pre-trained full modal output model based on full modal input; in response to receiving model acquisition information for a target business scenario, determining an input modality set and business output information corresponding to the target business scenario; for each at least one input modality in the input modality set, performing the following first generation step: based on the at least one input modality, performing model pruning on the full modal output model to generate a pruned multimodal output model, wherein the modality of the input data corresponding to the pruned multimodal output model is the at least one input modality, and wherein the pruned multimodal The model parameters corresponding to the modal output model are the same as the model parameters of the sub-modal output model with the same corresponding model structure in the above-mentioned full modal output model; a multimodal data set corresponding to the above-mentioned at least one input modality is obtained; multimodal data preprocessing is performed on the above-mentioned multimodal data set to generate a preprocessed data set; the above-mentioned preprocessed data set is used as a training data set, and according to the business output information, the above-mentioned trimmed multimodal output model is trained to generate a trained multimodal output model; the obtained at least one trained multimodal output model is stored as a model, and the at least one trained multimodal output model is deployed in the node cluster corresponding to the above-mentioned target business scenario.

[0009] In a second aspect, some embodiments of the present disclosure provide a model storage device, comprising: an acquisition unit, configured to acquire a pre-trained full modal output model based on full modal input; a determination unit, configured to determine, in response to receiving model acquisition information for a target business scenario, an input modality set and business output information corresponding to the target business scenario; an execution unit, configured to, for each at least one input modality in the input modality set, execute the following first generation step: based on the at least one input modality, perform model pruning on the full modal output model to generate a pruned multimodal output model, wherein the modality of the input data corresponding to the pruned multimodal output model is the at least one input modality, and In the embodiment, the model parameters corresponding to the trimmed multimodal output model are the same as the model parameters of the sub-modal output model with the same corresponding model structure in the full modal output model; a multimodal data set corresponding to the at least one input modality is obtained; multimodal data preprocessing is performed on the multimodal data set to generate a preprocessed data set; the preprocessed data set is used as a training data set, and according to the business output information, the trimmed multimodal output model is trained to generate a trained multimodal output model; a storage unit is configured to store the obtained at least one trained multimodal output model, and deploy the at least one trained multimodal output model in a node cluster corresponding to the target business scenario.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the model storage method of some embodiments of the present disclosure, the model landing and storage for the target business scenario can be accurately and efficiently realized, so as to facilitate the use of the model in the subsequent target business scenario. Specifically, the reason why the model landing of the relevant target business scenario is not accurate and efficient is that: for the predetermined business scenario with massive demand, targeted model structure design is required, which will waste a lot of model design resources, resulting in a long landing time of the business scenario model, which is not convenient for the use of the model in the business scenario. Based on this, the model storage method of some embodiments of the present disclosure, first, obtains a pre-trained full modal output model based on full modal input as the total model under each full modality, and by considering various business scenarios with different modal outputs, the model can be output through the full model to achieve the corresponding acquisition of the model structure, and improve the model landing efficiency in the target business scenario while ensuring the matching of the model structure. Then, in response to receiving the model acquisition information for the target business scenario, the input modality set and business output information corresponding to the above target business scenario are determined. Here, through the input modality set and business output information corresponding to the target business scenario, it is convenient to tailor the subsequent model to adapt to the model application in the target business scenario. Then, for each at least one input modality in the above input modality set, the following first generation step is performed: the first step is to perform model tailoring on the above full modality output model according to the above at least one input modality to quickly and accurately generate a tailored multimodal output model. Among them, the modality of the input data corresponding to the above-mentioned tailored multimodal output model is the above-mentioned at least one input modality, wherein the model parameters corresponding to the above-mentioned tailored multimodal output model are the same as the model parameters of the sub-modal output model with the same corresponding model structure in the above-mentioned full modality output model. Here, the model of the full modality output model can be effectively tailored by at least one input modality to meet the scenario requirements of the target business scenario. The second step is to obtain a multimodal data set corresponding to the above at least one input modality for model training of the tailored multimodal output model. The third step is to perform multimodal data preprocessing on the above multimodal data set to generate a preprocessing data set to generate a high-quality training data set. In the fourth step, the preprocessed data set is used as a training data set, and the trimmed multimodal output model is trained according to the business output information to generate a trained multimodal output model, so as to obtain a trained output model for at least one input modality and a target business scenario. Finally, the at least one trained multimodal output model is stored for model scheduling in the case of a usage scenario corresponding to the target business scenario, and the at least one trained multimodal output model is deployed in the node cluster corresponding to the target business scenario.In summary, based on the input modality set and business output information of the target business scenario, by tailoring the corresponding full modality output model under the corresponding input modality, a multimodal output model for each at least one input modality used in the target business scenario can be quickly obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the model storage method according to the present disclosure;

[0015] Figure 2 is a schematic diagram of the structure of some embodiments of the model storage device according to the present disclosure;

[0016] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] refer to Figure 1 , shows a process 100 of some embodiments of the model storage method according to the present disclosure. The model storage method comprises the following steps:

[0024] Step 101, obtaining a pre-trained full modality output model based on full modality input.

[0025] In some embodiments, the execution subject of the above-mentioned model storage method can obtain a pre-trained full modality output model based on full modality input through a wired connection or a wireless connection. Among them, the full modality output model based on full modality input can be a multimodal basic neural network model whose input is data under each modality and whose output can also be data under each modality. That is, the full modality output model based on full modality input can input multimodal data and can also output multimodal output results. The full modality input can be the data corresponding to each full modality under the full modality set as input. The full modality set may include: audio modality, text modality and image modality. In practice, the full modality output model may include: multiple parallel connected modality feature extraction models, multimodal feature fusion models, and output layers based on the full modality set. The modality feature extraction model may be a neural network model that extracts feature information for data under the corresponding modality. For example, for image modality data, the modality feature extraction model may be a multi-layer series convolutional neural network model. The multimodal feature fusion model may be a neural network model that fuses feature information of each feature information under multiple modalities. In practice, the multimodal feature fusion model may be a self-attention mechanism model. The business output information generation layer set based on the full modal set may be a network layer that generates business output information of each full modality in the full modal set. There is a one-to-one correspondence between the modalities in the full modal set and the business output information generation layers in the business output information generation layer set.

[0026] Step 102, in response to receiving the model acquisition information for the target business scenario, determining the input modality set and business output information corresponding to the target business scenario.

[0027] In some embodiments, in response to receiving model acquisition information for a target business scenario, the above-mentioned execution entity may determine the input modality set and business output information corresponding to the above-mentioned target business scenario. Among them, the target business scenario may be a business scenario to be applied to the model. For example, the target business scenario may be an e-commerce business scenario. The model acquisition information may be information for acquiring a model. For example, the model acquisition information may be a model acquisition request. The input modality set may be various data modalities of a data set input in the target business scenario. The business output information may represent the business goal to be output in the target business scenario. For example, the business output information may be a user sales forecast business goal.

[0028] Step 103: for each at least one input modality in the input modality set, perform the following first generation step:

[0029] Step 1031: prune the full modal output model according to the at least one input modality to generate a pruned multimodal output model.

[0030] In some embodiments, the execution subject may perform model tailoring on the full modal output model according to the at least one input modality to generate a tailored multimodal output model. The modality of the input data corresponding to the tailored multimodal output model is the at least one input modality, and the model parameters corresponding to the tailored multimodal output model are the same as the model parameters of the submodal output model with the same corresponding model structure in the full modal output model. That is, the submodal output model has the same model structure as the tailored multimodal output model, and the corresponding model parameters are the same.

[0031] In some optional implementations of some embodiments, the above-mentioned full modal output model includes: multiple parallel-connected modal feature extraction models, a multimodal feature fusion model, and a business output information generation layer set based on the full modal set. Among them, the modal feature extraction models in the multiple parallel-connected modal feature extraction models have a one-to-one correspondence with the modalities in the full modal set. The multimodal feature fusion model can be a network layer that performs feature fusion of at least one modal feature extraction information. The business output information generation layer in the business output information generation layer set has a one-to-one correspondence with the modality in the full modal set. The number of layers of the network extraction layer corresponding to each modal feature extraction model depends on the corresponding modality. For example, the number of layers of the network extraction layer corresponding to the image modality is 101 layers.

[0032] Optionally, the step of performing model pruning on the full modality output model according to the at least one input modality to generate a pruned multimodal output model may include the following steps:

[0033] The first step is to determine at least one difference mode between the at least one input mode and the full modality set, wherein the at least one difference mode may be at least one mode that is different between each mode in the at least one input mode and each mode in the full modality set.

[0034] In the second step, the modal feature extraction model and the business output information generation layer corresponding to at least one difference modality are removed from the above-mentioned full modal output model to generate the removed full modal output model as the pruned multimodal output model.

[0035] Step 1032: Obtain a multimodal data set corresponding to the at least one input modality.

[0036] In some embodiments, the execution subject may obtain a multimodal data set corresponding to the at least one input modality. The multimodal data in the multimodal data set includes: at least one input modality data corresponding to the at least one input modality. There is a one-to-one correspondence between the input modality in the at least one input modality and the input modality data in the at least one input modality data.

[0037] Step 1033: perform multimodal data preprocessing on the multimodal data set to generate a preprocessed data set.

[0038] In some embodiments, the execution subject may perform multimodal data preprocessing on the multimodal data set to generate a preprocessed data set, wherein the multimodal data preprocessing may include: missing modal data supplementation and data normalization.

[0039] Step 1034, using the preprocessed data set as a training data set, and performing model training on the trimmed multimodal output model according to the business output information to generate a trained multimodal output model.

[0040] In some embodiments, the execution subject may use the preprocessed data set as a training data set, and perform model training on the trimmed multimodal output model according to the business output information to generate a trained multimodal output model. The trained multimodal output model may be a model after model training is completed.

[0041] In some optional implementations of some embodiments, the above-mentioned preprocessed data set is used as a training data set, and according to the business output information, the above-mentioned cropped multimodal output model is trained to generate a trained multimodal output model, which may include the following steps:

[0042] In the first step, first target training data is selected from the first training data set, wherein the first training data includes: modal input data under at least one input modality, wherein the input content corresponding to the modal input data is input content under at least one input modality.

[0043] In the second step, for the first target training data, perform the following first training steps:

[0044] Sub-step 1: splitting the first target training data into data modalities to generate at least one first modal input data, wherein the modalities corresponding to the first modal input data in the at least one first modal input data are all different.

[0045] Sub-step 2: for each first modal input data in at least one first modal input data, perform the following second generating step:

[0046] In the first sub-step, the first modal input data is input into the corresponding modal feature extraction model included in the trimmed multimodal output model to generate modal feature information. The modal feature extraction model can be a neural network model for extracting modal feature information. The model structure corresponding to the modal feature extraction model can be set to a corresponding network structure according to the different corresponding modalities.

[0047] In a second sub-step, the first modal input data is input into a first deep-level feature extraction model of multiple layers in series to generate first deep-level feature extraction information. The first deep-level feature extraction model may be a neural network model for deep-level feature extraction. For example, the first deep-level feature extraction model may be a convolutional neural network model.

[0048] In a third sub-step, the first deep-level feature extraction information is input into a target business output information generation layer to generate candidate business output information corresponding to a second target modality. The second target modality is the same as the modality corresponding to the modality input data. The target business output information generation layer is a business output information generation layer corresponding to the second target modality. The target business output information generation layer may be a network layer that generates business output information. For example, the target business output information generation layer may be a multi-layer fully connected layer.

[0049] Sub-step 3: inputting the obtained at least one modal feature information into the multimodal feature fusion model included in the pruned multimodal output model to generate multimodal feature fusion information.

[0050] Sub-step 4: input the multimodal feature fusion information into a second deep-level feature extraction model with multiple layers connected in series to generate second deep-level feature extraction information. The second deep-level feature extraction model may be a convolutional neural network model for extracting deep-level feature information.

[0051] Sub-step 5, inputting the second deep-level feature extraction information into at least one business output information generation layer to output at least one business output information. Wherein, the business output information generation layer in the at least one business output information generation layer has a one-to-one correspondence with the input modality in the at least one input modality. The business output information in the at least one business output information has a one-to-one correspondence with the input modality in the at least one input modality.

[0052] Sub-step 6: Generate first loss information representing the difference between the information bodies of at least one business output information. The first loss information may represent the difference between at least one information body corresponding to at least one business output information.

[0053] As an example, the execution entity may use a cross entropy loss function to generate first loss information that characterizes the information body difference between at least one business output information.

[0054] Sub-step 7: Generate second loss information according to the at least one service output information and the actual service output information, wherein the second loss information may represent the information difference between each service output information in the at least one service output information and the actual service output information.

[0055] As an example, the execution entity may generate the second loss information according to at least one business output information and actual business output information based on the first preset loss function.

[0056] Sub-step 8, generating at least one third loss information according to at least one business output information and at least one candidate business output information obtained. There is a one-to-one correspondence between the third loss information in the at least one third loss information and the candidate business output information in the at least one candidate business output information. The third loss information can represent the information difference between the business output information and the corresponding candidate business output information.

[0057] As an example, the execution entity may generate at least one third loss information according to at least one business output information and at least one obtained candidate business output information based on the second preset loss function.

[0058] Sub-step 9, determining whether the pruned multimodal output model is trained based on the first loss information, the second loss information and at least one third loss information.

[0059] As an example, in response to determining that the first loss information is less than the first value, the second loss information is less than the second value, and each third loss information is less than the third value, it is determined that the training of the pruned multimodal output model is completed. In response to determining that the first loss information is greater than or equal to the first value, the second loss information is greater than or equal to the second value, and each third loss information is greater than or equal to at least one of the third values, it is determined that the training of the pruned multimodal output model is not completed.

[0060] Sub-step 10, in response to determining that the training is completed, determining the pruned multimodal output model as the trained multimodal output model.

[0061] In the third step, in response to determining that the training is not completed, the model parameters of the pruned multimodal output model are updated according to the first loss information, the second loss information and at least one third loss information to generate a first updated multimodal output model.

[0062] As an example, the above-mentioned execution entity can update the model parameters of the pruned multimodal output model by back propagation according to the first loss information, the second loss information and at least one third loss information to generate a first updated multimodal output model.

[0063] In the fourth step, the first target training data is removed from the first training data set to generate a first training data set after removal.

[0064] The fifth step is to select the first candidate training data from the first training data set after removal.

[0065] In the sixth step, the first candidate training data is used as the first target training data, and the first updated multimodal output model is used as the pruned multimodal output model, and the first training step is continued.

[0066] Optionally, the generating of the first loss information characterizing the information main body difference between at least one service output information may include the following steps:

[0067] In the first step, the target semantic extraction model is used to extract the semantic information corresponding to each of the at least one business output information to generate at least one semantic information. The target semantic extraction model may be a neural network model for extracting the semantic information corresponding to the business output information. In practice, the target semantic extraction model may include: at least one semantic extraction model corresponding to at least one modality. For example, for the image modality, the semantic extraction model may be an image text extraction model.

[0068] In the second step, a first candidate loss information for at least one semantic information is determined using a first loss function. The first candidate loss information may represent a semantic difference between each semantic information in the at least one semantic information. The first loss function may be a cross entropy loss function.

[0069] In the third step, the target keyword extraction model is used to extract the keyword set corresponding to each business output information in the at least one business output information to generate at least one keyword set. The target keyword extraction model may be a neural network model for extracting keywords.

[0070] In the fourth step, the second loss function is used to determine second candidate loss information for at least one keyword set, wherein the second candidate loss information may represent the word set differences between the keyword sets in the at least one keyword set.

[0071] In the fifth step, the weighted loss information corresponding to the first candidate loss information and the second candidate loss information is determined as the first loss information.

[0072] In some optional implementations of some embodiments, the at least one input modality includes: an image input modality, a text input modality, and an audio input modality.

[0073] Optionally, the preprocessed data set is used as a training data set, and model training is performed on the pruned multimodal output model according to the business output information to generate a trained multimodal output model, including:

[0074] The first step is to select second target training data from the second training data set, wherein the second training data includes: modal input data under at least one input modality.

[0075] In the second step, for the second target training data, the following second training steps are performed:

[0076] Sub-step 1: performing data modality splitting on the second target training data to obtain image input modality data, text input modality data and audio input modality data.

[0077] Sub-step 2, inputting the image input modality data into the image modality feature information extraction model included in the cropped multimodal output model to generate image modality feature information. The image modality feature information extraction model can be a neural network model that extracts feature information under the image modality. The image modality feature information extraction model can be a multi-layer series convolutional neural network model.

[0078] Sub-step 3, inputting the text input modality data into the text input modality feature information extraction model included in the trimmed multimodal output model to generate text modality feature information. The text modality feature information extraction model can be a neural network model that extracts feature information under the text modality. The text modality feature information extraction model can be a recurrent neural network model.

[0079] Sub-step 4: input the image modality feature information and the text modality feature information into a first fused feature information generation model to generate first fused feature information. The first fused feature information generation model may be a neural network model that fuses feature information of multiple input modalities.

[0080] Sub-step 5: Input the first fusion feature information into a third deep-level feature extraction model to generate third deep-level feature extraction information. The third deep-level feature extraction model may be a neural network model for deep-level feature extraction. The third deep-level feature extraction model may be a multi-layer series convolutional neural network model.

[0081] Sub-step 6: input the third deep-level feature extraction information into the image-modality-based business output information generation layer to generate a business output image.

[0082] Sub-step 7: input the third deep-level feature extraction information into a text-based business output information generation layer to generate a business output text.

[0083] Sub-step 8: Input the audio input modality data and the first fused feature information into a second fused feature information generation model to generate second fused feature information. The second fused feature information generation model may be a neural network model that fuses feature information of multiple input modalities.

[0084] Sub-step 9: input the second fused feature information into a fourth deep-level feature extraction model to generate fourth deep-level feature extraction information. The fourth deep-level feature extraction model may be a neural network model for deep-level feature extraction. The fourth deep-level feature extraction model may be a multi-layer series convolutional neural network model.

[0085] Sub-step 10: input the fourth deep-level feature extraction information into the business output information generation layer based on the audio modality to generate business output audio.

[0086] Sub-step 11: Generate fourth loss information for the service output image, the service output text, and the service output audio, wherein the fourth loss information may represent content difference information between the service output image, the service output text, and the service output audio.

[0087] As an example, the execution subject may perform modal unification on the service output image, the service output text, and the service output audio to unify them into a text modal form to generate a text modal content information set. Then, using a text loss function, the information difference information between each text modal content information in the text modal content information set is determined.

[0088] Sub-step 12, generating fifth loss information based on the business output image, the actual business output information, the business output text and the business output audio. The fifth loss information can represent the information difference between the actual business output information and the business output image, the business output text and the business output audio. The generation of the fifth loss information can refer to the generation of the fourth loss information.

[0089] Sub-step 13, determining whether the pruned multimodal output model is trained based on the fourth loss information and the fifth loss information.

[0090] As an example, in response to determining that the fourth loss information is less than the fourth value and the fifth loss information is less than the fifth value, it is determined that the training of the above-mentioned pruned multimodal output model is completed. In response to determining that the fourth loss information is greater than or equal to the fourth value or the fifth loss information is greater than or equal to the fifth value, it is determined that the training of the above-mentioned pruned multimodal output model is not completed.

[0091] Sub-step 14, in response to determining that the training is completed, determining the pruned multimodal output model as the trained multimodal output model.

[0092] In the third step, in response to determining that the training is not completed, the model parameters of the pruned multimodal output model are updated according to the fourth loss information and the fifth loss information to obtain a second updated multimodal output model.

[0093] As an example, the execution entity may update the model parameters of the pruned multimodal output model by back propagation according to the fourth loss information and the fifth loss information to obtain a second updated multimodal output model.

[0094] The fourth step is to remove the second target training data from the second training data set to obtain the removed second training data set.

[0095] The fifth step is to select the second candidate training data from the removed second training data set.

[0096] In the sixth step, the second candidate training data is used as the second target training data, and the second updated multimodal output model is used as the pruned multimodal output model, and the second training step is continued.

[0097] Optionally, the determining whether the pruned multimodal output model is trained according to the fourth loss information and the fifth loss information includes:

[0098] The first step is to generate a predicted service output image for the above-mentioned image modality feature information, wherein the predicted service output image is service information in the form of an image output based on the feature semantic content corresponding to the image modality feature information.

[0099] The second step is to generate a predicted service output text for the above text modal feature information, wherein the predicted service output text is service information in text form output based on the feature semantic content corresponding to the text modal feature information.

[0100] The third step is to generate a predicted service output audio for the above audio modal feature information, wherein the predicted service output audio is service information in the form of audio output based on the feature semantic content corresponding to the text modal feature information.

[0101] The fourth step is to generate sixth loss information according to the predicted service output image and the service output image, wherein the sixth loss information may represent the image content difference between the predicted service output image and the service output image.

[0102] Step 5: Generate seventh loss information based on the predicted business output text and the business output text, wherein the seventh loss information may represent the text content difference between the predicted business output text and the business output text.

[0103] Step 6: Generate eighth loss information based on the predicted service output audio and the service output audio, wherein the eighth loss information may represent the difference in audio content between the predicted service output audio and the service output audio.

[0104] The seventh step is to determine the first feature similarity information between the above-mentioned image modal feature information and the above-mentioned first fused feature information, and to determine the second feature similarity information between the text modal feature information and the above-mentioned first fused feature information. The first feature similarity information can characterize the feature difference between the above-mentioned image modal feature information and the above-mentioned first fused feature information. The second feature similarity information can characterize the feature difference between the above-mentioned text modal feature information and the above-mentioned first fused feature information. The first feature similarity information and the second feature similarity information can be values ​​between 0 and 1.

[0105] Step 8: Generate ninth loss information for the first feature similarity information and generate tenth loss information for the second feature similarity information. The ninth loss information may represent the numerical difference information between the first feature similarity information and the target similarity information. The tenth loss information may represent the numerical difference information between the second feature similarity information and the target similarity information.

[0106] The ninth step is to determine the third feature similarity information between the audio modality feature information and the second fusion feature information, wherein the third feature similarity information may represent feature difference information between the audio modality feature information and the second fusion feature information.

[0107] In the tenth step, the eleventh loss information is generated for the third characteristic similarity information, wherein the eleventh loss information can represent the value difference information between the eleventh loss information and the target value.

[0108] In the eleventh step, it is determined whether the training of the pruned multimodal output model is completed according to the fourth loss information, the fifth loss information, the sixth loss information, the seventh loss information, the eighth loss information, the ninth loss information, the tenth loss information and the eleventh loss information. The specific implementation method will not be repeated here.

[0109] Step 104: store the obtained at least one trained multimodal output model, and deploy the at least one trained multimodal output model in a node cluster corresponding to the target business scenario.

[0110] In some embodiments, the execution entity may store the at least one trained multimodal output model obtained, and deploy the at least one trained multimodal output model in a node cluster corresponding to the target business scenario.

[0111] As an example, the execution entity may store the obtained at least one trained multimodal output model in a target container.

[0112] In some optional implementations of some embodiments, deploying the at least one trained multimodal output model on a node cluster corresponding to the target business scenario includes:

[0113] The first step is to determine the peak hours of the target business scenario. For example, if the target business scenario is click-through rate prediction, the peak hours are {11 am to 2 pm, 6 pm to 12 am}.

[0114] As an example, the above-mentioned execution entity can determine the peak period corresponding to the target business scenario by means of peak period query.

[0115] In the second step, for each compute node in the node cluster, perform the following processing steps:

[0116] Sub-step 1, determining the minimum amount sequence of historical online available resources of the computing node in the peak period of the scenario in the historical time period. The minimum amount of historical online available resources can be the minimum value of online resources available in the peak period of the scenario at the target historical time.

[0117] Sub-step 2: averaging each of the historical online minimum available resource quantities in the above sequence of historical online minimum available resource quantities to generate an average available resource quantity.

[0118] The third step is to obtain the maximum online resource amount required during the peak period for each multimodal output model in the at least one trained multimodal output model, and obtain at least one maximum online resource amount required.

[0119] In the fourth step, according to the obtained average available resource set and at least one required maximum online resource, under the target constraint condition, the deployment distribution information between the computing nodes and the trained multimodal output model is generated. The target constraint condition may be that the sum of the average available resource corresponding to at least one allocated computing node is greater than the sum of the required maximum online resource corresponding to at least one trained multimodal output model.

[0120] The fifth step is to deploy the at least one trained multimodal output model in the node cluster corresponding to the target business scenario according to the deployment distribution information.

[0121] In some optional implementations of some embodiments described above, as an inventive point of the present disclosure, the technical problem "After obtaining at least one multimodal model for the target business scenario, how to deploy the model during the peak period of the target business scenario to avoid the problem of long feedback time in the target business scenario caused by insufficient request resources during online prediction has become an important concern at present." Based on this, the present disclosure determines the average available resource amount corresponding to each computing node in the node cluster, and determines the maximum online resource amount required for each trained multimodal output model. Under the constraints of the target constraints, the deployment distribution information can be accurately generated to ensure that the target business will not encounter the problem of insufficient request resources during online prediction during the peak period.

[0122] In some optional implementations of some embodiments, after step 104, the steps further include:

[0123] The first step is, in response to receiving input data for the target business scenario, determining a modality corresponding to the input data as a first target modality, and determining an output modality form for the target business scenario.

[0124] In the second step, according to the above modality information, a trained multimodal output model corresponding to the same input modality as the above first target modality is obtained from at least one deployed trained multimodal output model as the target trained multimodal output model. The modality of the input data corresponding to the target trained multimodal output model is the first target modality.

[0125] The third step is to input the above input data into the multimodal output model after the above target training to generate the output result in the above output modality.

[0126] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the model storage method of some embodiments of the present disclosure, the model landing and storage for the target business scenario can be accurately and efficiently realized, so as to facilitate the use of the model in the subsequent target business scenario. Specifically, the reason why the model landing of the relevant target business scenario is not accurate and efficient is that: for the predetermined business scenario with massive demand, targeted model structure design is required, which will waste a lot of model design resources, resulting in a long landing time of the business scenario model, which is not convenient for the use of the model in the business scenario. Based on this, the model storage method of some embodiments of the present disclosure, first, obtains a pre-trained full modal output model based on full modal input as the total model under each full modality, and by considering various business scenarios with different modal outputs, the model can be output through the full model to achieve the corresponding acquisition of the model structure, and improve the model landing efficiency in the target business scenario while ensuring the matching of the model structure. Then, in response to receiving the model acquisition information for the target business scenario, the input modality set and business output information corresponding to the above target business scenario are determined. Here, through the input modality set and business output information corresponding to the target business scenario, it is convenient to tailor the subsequent model to adapt to the model application in the target business scenario. Then, for each at least one input modality in the above input modality set, the following first generation step is performed: the first step is to perform model tailoring on the above full modality output model according to the above at least one input modality to quickly and accurately generate a tailored multimodal output model. Among them, the modality of the input data corresponding to the above-mentioned tailored multimodal output model is the above-mentioned at least one input modality, wherein the model parameters corresponding to the above-mentioned tailored multimodal output model are the same as the model parameters of the sub-modal output model with the same corresponding model structure in the above-mentioned full modality output model. Here, the model of the full modality output model can be effectively tailored by at least one input modality to meet the scenario requirements of the target business scenario. The second step is to obtain a multimodal data set corresponding to the above at least one input modality for model training of the tailored multimodal output model. The third step is to perform multimodal data preprocessing on the above multimodal data set to generate a preprocessing data set to generate a high-quality training data set. In the fourth step, the preprocessed data set is used as a training data set, and the trimmed multimodal output model is trained according to the business output information to generate a trained multimodal output model, so as to obtain a trained output model for at least one input modality and a target business scenario. Finally, the at least one trained multimodal output model is stored for model scheduling in the case of a usage scenario corresponding to the target business scenario, and the at least one trained multimodal output model is deployed in the node cluster corresponding to the target business scenario.In summary, based on the input modality set and business output information of the target business scenario, by tailoring the corresponding full modality output model under the corresponding input modality, a multimodal output model for each at least one input modality used in the target business scenario can be quickly obtained.

[0127] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a model storage device, and these device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the model storage device can be specifically applied to various electronic devices.

[0128] like Figure 2 As shown, a model storage device 200 includes: an acquisition unit 201, a determination unit 202, an execution unit 203 and a storage unit 204. The acquisition unit 201 is configured to acquire a pre-trained full modal output model based on full modal input; the determination unit 202 is configured to determine the input modal set and business output information corresponding to the target business scenario in response to receiving model acquisition information for the target business scenario; the execution unit 203 is configured to perform the following first generation step for each at least one input modality in the input modality set: according to the at least one input modality, the full modal output model is model pruned to generate a pruned multimodal output model, wherein the modality of the input data corresponding to the pruned multimodal output model is the at least one input modality, wherein the pruned multimodal The model parameters corresponding to the modal output model are the same as the model parameters of the sub-modal output model with the same corresponding model structure in the full modal output model; a multimodal data set corresponding to the at least one input modality is obtained; multimodal data preprocessing is performed on the multimodal data set to generate a preprocessed data set; the preprocessed data set is used as a training data set, and according to the business output information, the trimmed multimodal output model is trained to generate a trained multimodal output model; the storage unit 204 is configured to store the obtained at least one trained multimodal output model, and deploy the at least one trained multimodal output model in the node cluster corresponding to the target business scenario.

[0129] It is understandable that the units recorded in the model storage device 200 and the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the model storage device 200 and the units contained therein, and will not be described in detail here.

[0130] Reference below Figure 3, which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0131] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0132] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0133] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0134] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0135] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0136] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being installed in the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a pre-trained full modal output model based on full modal input; in response to receiving model acquisition information for the target business scenario, determines the input modality set and business output information corresponding to the above-mentioned target business scenario; for each at least one input modality in the above-mentioned input modality set, executes the following first generation step: according to the above-mentioned at least one input modality, the above-mentioned full modal output model is model-pruned to generate a pruned multi-modal output model, wherein the modality of the input data corresponding to the above-mentioned pruned multi-modal output model is the above-mentioned at least one input modality, Among them, the model parameters corresponding to the above-mentioned trimmed multimodal output model are the same as the model parameters of the sub-modal output model with the same corresponding model structure in the above-mentioned full modal output model; a multimodal data set corresponding to the above-mentioned at least one input modality is obtained; multimodal data preprocessing is performed on the above-mentioned multimodal data set to generate a preprocessed data set; the above-mentioned preprocessed data set is used as a training data set, and according to the business output information, the model training is performed on the above-mentioned trimmed multimodal output model to generate a trained multimodal output model; the obtained at least one trained multimodal output model is model stored, and the at least one trained multimodal output model is deployed in the node cluster corresponding to the above-mentioned target business scenario.

[0137] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising an acquisition unit, a determination unit, an execution unit, and a storage unit. Among them, the names of these units do not constitute a limitation on the unit itself in certain circumstances, for example, the acquisition unit may also be described as "a unit for acquiring a pre-trained full modal output model based on full modal input".

[0140] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0141] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A model storage method, comprising: Obtain a pre-trained full modal output model based on full modal input; In response to receiving the model acquisition information for the target business scenario, determining the input modality set and business output information corresponding to the target business scenario; For each at least one input modality in the input modality set, the following first generating step is performed: According to the at least one input modality, the full modality output model is model pruned to generate a pruned multimodal output model, wherein the modality of the input data corresponding to the pruned multimodal output model is the at least one input modality, and wherein the model parameters corresponding to the pruned multimodal output model are the same as the model parameters of the submodal output model having the same corresponding model structure in the full modality output model; Acquire a multimodal data set corresponding to the at least one input modality; Performing multimodal data preprocessing on the multimodal data set to generate a preprocessed data set; Using the preprocessed data set as a training data set, and performing model training on the pruned multimodal output model according to the business output information to generate a trained multimodal output model; The at least one trained multimodal output model obtained is stored in a model, and the at least one trained multimodal output model is deployed in a node cluster corresponding to the target business scenario.

2. The method according to claim 1, wherein: The method further comprises: In response to receiving input data for the target business scenario, determining a modality corresponding to the input data as a first target modality, and determining an output modality form for the target business scenario; According to the modality information, obtaining a trained multimodal output model having a corresponding input modality identical to the first target modality from at least one deployed trained multimodal output model as a target trained multimodal output model; The input data is input into the target trained multimodal output model to generate an output result in the output modality.

3. The method according to claim 1, wherein: The full modal output model includes: a plurality of parallel connected modal feature extraction models, a multimodal feature fusion model, and a layer set generated based on the business output information of the full modal set; and The step of performing model pruning on the full modality output model according to the at least one input modality to generate a pruned multimodal output model includes: Determining at least one difference modality between the at least one input modality and the full set of modalities; The modal feature extraction model and the business output information generation layer corresponding to at least one difference modality are removed from the full modal output model to generate a removed full modal output model as a pruned multimodal output model.

4. The method according to claim 1, wherein: The using the preprocessed data set as a training data set and performing model training on the trimmed multimodal output model according to the business output information to generate a trained multimodal output model includes: Selecting first target training data from a first training data set, wherein the first training data includes: modal input data under at least one input modality; For the first target training data, perform the following first training steps: Performing data modality splitting on the first target training data to generate at least one first modality input data; For each first modality input data of at least one first modality input data, the following second generating step is performed: Inputting the first modal input data into a corresponding modal feature extraction model included in the pruned multimodal output model to generate modal feature information; Inputting the first modal input data into a first deep-level feature extraction model of multiple layers in series to generate first deep-level feature extraction information; Inputting the first deep-level feature extraction information into a target business output information generation layer to generate candidate business output information corresponding to a second target modality, wherein the second target modality is the same as the modality corresponding to the modality input data, and the target business output information generation layer is a business output information generation layer corresponding to the second target modality; Inputting the obtained at least one modal feature information into a multimodal feature fusion model included in the pruned multimodal output model to generate multimodal feature fusion information; Inputting the multimodal feature fusion information into a multi-layer series-connected second deep-level feature extraction model to generate second deep-level feature extraction information; Inputting the second deep-level feature extraction information into at least one business output information generation layer to output at least one business output information, wherein the business output information generation layer in the at least one business output information generation layer has a one-to-one correspondence with the input modality in the at least one input modality, and the business output information in the at least one business output information has a one-to-one correspondence with the input modality in the at least one input modality; Generate first loss information representing a difference in the main body of information between at least one business output information; generating second loss information according to at least one business output information and actual business output information; generating at least one third loss information according to at least one service output information and at least one obtained candidate service output information; Determining whether the pruned multimodal output model is trained according to the first loss information, the second loss information, and at least one third loss information; In response to determining that the training is complete, determining the pruned multimodal output model as the trained multimodal output model; In response to determining that the training is not completed, updating model parameters of the pruned multimodal output model according to the first loss information, the second loss information, and at least one third loss information to generate a first updated multimodal output model; Removing the first target training data from the first training data set to generate a first training data set after removal; Selecting first candidate training data from the removed first training data set; The first candidate training data is used as the first target training data, and the first updated multimodal output model is used as the pruned multimodal output model, and the first training step is continued.

5. The method according to claim 4, wherein: The generating of the first loss information characterizing the information main body difference between at least one business output information includes: Extracting semantic information corresponding to each piece of business output information in the at least one business output information using a target semantic extraction model to generate at least one piece of semantic information; Determine, using a first loss function, first candidate loss information for at least one semantic information; Using the target keyword extraction model, extracting a keyword set corresponding to each business output information in the at least one business output information to generate at least one keyword set; Determining second candidate loss information for at least one keyword set using a second loss function; The weighted loss information corresponding to the first candidate loss information and the second candidate loss information is determined as the first loss information.

6. The method according to claim 1, wherein: The at least one input modality includes: an image input modality, a text input modality, and an audio input modality; and The using the preprocessed data set as a training data set and performing model training on the trimmed multimodal output model according to the business output information to generate a trained multimodal output model includes: Selecting second target training data from the second training data set, wherein the second training data includes: modal input data under at least one input modality; For the second target training data, perform the following second training steps: Performing data modality splitting on the second target training data to obtain image input modality data, text input modality data, and audio input modality data; Inputting the image input modality data into an image modality feature information extraction model included in the cropped multimodal output model to generate image modality feature information; Inputting the text input modality data into the text input modality feature information extraction model included in the trimmed multimodal output model to generate text modality feature information; Inputting the image modality feature information and the text modality feature information into a first fused feature information generation model to generate first fused feature information; Inputting the first fused feature information into a third deep-level feature extraction model to generate third deep-level feature extraction information; Inputting the third deep-level feature extraction information into the image-modality-based business output information generation layer to generate a business output image; Inputting the third deep-level feature extraction information into a text-based business output information generation layer to generate a business output text; Inputting the audio input modality data and the first fused feature information into a second fused feature information generation model to generate second fused feature information; Inputting the second fused feature information into a fourth deep-level feature extraction model to generate fourth deep-level feature extraction information; Inputting the fourth deep-level feature extraction information into the business output information generation layer based on the audio modality to generate business output audio; generating fourth loss information for the service output image, the service output text, and the service output audio; generating fifth loss information according to the service output image, the actual service output information, the service output text and the service output audio; Determining whether the pruned multimodal output model has been trained according to the fourth loss information and the fifth loss information; In response to determining that the training is complete, determining the pruned multimodal output model as the trained multimodal output model; In response to determining that the training is not completed, updating the model parameters of the pruned multimodal output model according to the fourth loss information and the fifth loss information to obtain a second updated multimodal output model; Removing the second target training data from the second training data set to obtain a second training data set after removal; Selecting second candidate training data from the removed second training data set; The second candidate training data is used as the second target training data, and the second updated multimodal output model is used as the pruned multimodal output model, and the second training step is continued.

7. The method according to claim 6, wherein: The determining, according to the fourth loss information and the fifth loss information, whether the pruned multimodal output model is trained, includes: Generating a predicted service output image for the image modality feature information; Generating a predicted business output text for the text modality feature information; Generating predicted service output audio for the audio modal feature information; generating sixth loss information according to the predicted business output image and the business output image; generating seventh loss information according to the predicted business output text and the business output text; generating eighth loss information according to the predicted service output audio and the service output audio; Determining first feature similarity information between the image modality feature information and the first fused feature information, and determining second feature similarity information between the text modality feature information and the first fused feature information; generating ninth loss information for the first feature similarity information, and generating tenth loss information for the second feature similarity information; Determining third feature similarity information between the audio modality feature information and the second fusion feature information; generating eleventh loss information for the third feature similarity information; According to the fourth loss information, the fifth loss information, the sixth loss information, the seventh loss information, the eighth loss information, the ninth loss information, the tenth loss information and the eleventh loss information, it is determined whether the training of the cropped multimodal output model is completed.

8. A model storage device, comprising: An acquisition unit configured to acquire a pre-trained full modality output model based on full modality input; A determination unit, configured to, in response to receiving model acquisition information for a target business scenario, determine an input modality set and business output information corresponding to the target business scenario; The execution unit is configured to perform the following first generation step for each at least one input modality in the input modality set: according to the at least one input modality, perform model pruning on the full modality output model to generate a pruned multimodal output model, wherein the modality of the input data corresponding to the pruned multimodal output model is the at least one input modality, wherein the model parameters corresponding to the pruned multimodal output model are the same as the model parameters of the submodal output model with the same corresponding model structure in the full modality output model; obtain a multimodal data set corresponding to the at least one input modality; perform multimodal data preprocessing on the multimodal data set to generate a preprocessed data set; use the preprocessed data set as a training data set, and perform model training on the pruned multimodal output model according to the business output information to generate a trained multimodal output model; The storage unit is configured to store the at least one trained multimodal output model obtained, and deploy the at least one trained multimodal output model in a node cluster corresponding to the target business scenario.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images