Model acceleration and data processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a model acceleration and data processing method and device, a storage medium and a program product. In the embodiment of the invention, the acceleration network driven by the information amount under the data perspective is provided, the neural network model is accelerated from the perspective of redundant data removal, and specifically, the acceleration network performs model acceleration on the neural network model by using an acceleration module; the acceleration module is located between the first calculation module and the second calculation module in the neural network model, and merges multiple pieces of mark information according to the information degree of the multiple pieces of mark information included in the first mark sequence output by the first calculation module, so that the calculation amount is reduced, and the calculation cost is reduced. Furthermore, the information degree comprises mutual information redundancy and semantic value information, the mutual information redundancy evaluates data repetition in the tagged sequence, and the semantic value information evaluates contribution of each piece of tagged information to overall semantics, so that in the merging process, the number of the tagged information is reduced, and the calculation quality of the model is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model acceleration and data processing method, device, storage medium and program product. Background Art

[0002] Multimodal large models (Vision Language Models, VLMs) are important components of artificial intelligence systems, covering the two major business scenarios of "understanding and generation", and are crucial for services such as retrieval recommendation, dialogue question and answer, and graphic content creation. With the continuous expansion of model parameters and data scale, multimodal large models have quickly achieved powerful performance in multiple fields, but at the cost of high computing costs, which hinders their application in real-world scenarios. Summary of the invention

[0003] Multiple aspects of the present application provide a model acceleration and data processing method, device, storage medium and program product to increase the computing speed of the neural network model, reduce the computing cost of the neural network model, and expand the application scenarios of the neural network model.

[0004] An embodiment of the present application provides a model acceleration method, which is applied to an acceleration network. The acceleration network uses an acceleration module to accelerate a neural network model. The acceleration module is located between a first computing module and a second computing module in the neural network model. The method includes: receiving a first label sequence output by a first computing module when executing a first model processing operation, the first label sequence including multiple label information; determining the information degree of at least part of the label information in the multiple label information based on the similarity between the multiple label information; merging at least part of the label information based on the information degree of at least part of the label information to obtain a second label sequence; and inputting the second label sequence into a second computing module so that the second computing module performs a second model processing operation based on the second label sequence.

[0005] An embodiment of the present application also provides a data processing method, which includes: responding to an access request of a target service, obtaining service demand information, the target service corresponding to a neural network model; based on the service demand information, calling the neural network model to perform model reasoning operations on the target service to obtain service result information of the target service; returning the service result information to the requesting end of the service request so that the requesting end can use the target service; wherein the neural network model includes at least a first computing module, a second computing module and an acceleration network; wherein the acceleration network uses the model acceleration method provided in the embodiment of the present application to perform model acceleration for the neural network model.

[0006] The embodiment of the present application also provides a data processing method, which includes: in response to a service request for a target service, obtaining service demand information, the target service corresponding to a multimodal generation model; based on the service demand information, calling the multimodal generation model to perform model reasoning operations on the target service to obtain service result information of the target service; returning the service result information to the request end of the service request, so that the request end can provide the target service; wherein the multimodal generation model includes at least a feedforward module, an attention module and an acceleration network, the acceleration network includes an acceleration module located before the target module and a restoration module located after the target module, and the target module is any one of the feedforward module and the attention module; wherein the acceleration network uses the model acceleration method provided in the embodiment of the present application to perform model acceleration for the multimodal generation model.

[0007] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the model acceleration method and data processing method provided in the embodiment of the present application.

[0008] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements each step in the model acceleration method and data processing method provided in the embodiment of the present application.

[0009] An embodiment of the present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the processor implements each step in the model acceleration method and data processing method provided in the embodiment of the present application.

[0010] In an embodiment of the present application, an acceleration network driven by information volume from a data perspective is provided to accelerate a neural network model from the perspective of removing redundant data. Specifically, the acceleration network uses an acceleration module to accelerate the neural network model. The acceleration module is located between a first computing module and a second computing module in the neural network model. For a first label sequence output by the first computing module, multiple label information is merged according to the information degree of the multiple label information included therein, thereby reducing the amount of calculation involved in subsequent model processing and reducing the calculation cost.

[0011] Furthermore, the information degree includes two dimensions: mutual information redundancy and semantic value information. Mutual information redundancy evaluates the data duplication in the tag sequence, and semantic value information evaluates the contribution of each tag information to the overall semantics. The tag sequence is merged from the two dimensions of mutual information redundancy and semantic value information, which reduces the amount of tag information while ensuring the quality of model calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 A flow chart of a model acceleration method provided for an exemplary embodiment of the present application;

[0014] Figure 2a An internal structure diagram of a neural network model provided for an exemplary embodiment of the present application;

[0015] Figure 2b An internal structure diagram of another neural network model provided for an exemplary embodiment of the present application;

[0016] Figure 3a A local structural diagram of a neural network model provided for an exemplary embodiment of the present application;

[0017] Figure 3b A schematic diagram of a data processing process for accelerating a network provided by an exemplary embodiment of the present application;

[0018] Figure 4 A schematic structural diagram of a model acceleration device provided for an exemplary embodiment of the present application;

[0019] Figure 5 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0022] The various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0023] With the continuous expansion of model parameters and data scale, various neural network models, especially multimodal large models, have rapidly achieved powerful performance in multiple fields, but at the cost of high computational costs, which hinders their application in real-world scenarios. To this end, it is necessary to accelerate the computational process of neural network models, increase the computational speed of neural network models, reduce the computational cost of neural network models, and expand the application scenarios of neural network models.

[0024] In some acceleration schemes, acceleration design is performed from the perspective of the neural network model, which is referred to as model-perspective acceleration schemes, such as model distillation, network pruning, floating-point quantization, etc.

[0025] Among them, distillation, pruning, and quantization can be collectively referred to as model-centric methods. Model-centric methods refer to techniques that improve the efficiency of the model by optimizing the model itself.

[0026] Distillation: Reduce model complexity and computational requirements by transferring the knowledge of the teacher model to the student model. During the distillation process, the student model learns the output and feature representation of the teacher model to achieve the goal of reducing computational and storage costs while maintaining performance. Among them, the teacher model is a well-trained, high-performance multimodal model. It is a complex model with multiple modal inputs (such as text, images, audio, etc.), usually containing multiple deep networks and a large number of parameters. The student model is a smaller and simpler multimodal model. The goal is to reduce computing resources and storage requirements while maintaining high performance. It usually has fewer parameters and a simpler structure, but is still able to handle multimodal inputs.

[0027] Pruning: By removing less important parameters or neurons in the model, the amount of computation and model size can be reduced, thereby speeding up the inference process and reducing memory consumption.

[0028] Quantization: Reduces computation and storage requirements while maintaining model performance as much as possible by reducing model weights and activation functions from high precision (such as 32-bit floating point) to low precision (such as 8-bit integer).

[0029] Although the above-mentioned acceleration solutions from the perspective of the model can improve the calculation speed of the neural network model to a certain extent, there are still some problems:

[0030] (1) Specificity: Model-based acceleration is developed only for specific architectures, which means that it is not compatible with various types of neural network models and has poor versatility. For example, the acceleration method for multimodal understanding models has poor effects in multimodal generation models and cannot be used universally.

[0031] (2) Retraining: To maintain high performance, model-based acceleration usually requires retraining or fine-tuning the neural network model, which inevitably consumes additional computing resources.

[0032] (3) Complexity: Model-based acceleration involves a large number of development skills, which increases the difficulty of application development.

[0033] In an embodiment of the present application, an acceleration scheme driven by information volume from a data perspective is provided to accelerate a neural network model from the perspective of removing redundant data. Specifically, the acceleration network uses an acceleration module to accelerate the neural network model. The acceleration module is located between the first computing module and the second computing module in the neural network model. For the first tag sequence output by the first computing module, multiple tag information is merged according to the information degree of the multiple tag information included therein, thereby reducing the amount of calculation involved in subsequent model processing, which can improve the calculation speed of the neural network model and reduce the calculation cost. In addition, compared with the acceleration scheme from a model perspective, the acceleration scheme driven by information volume from a data perspective provided in the embodiment of the present application is compatible with various neural network models, has high versatility, and does not require the neural network model to be retrained, which can save additional computing resources consumed by retraining the model, and can be imperceptible to the application without affecting the development and operation of the application.

[0034] Furthermore, the information degree includes two dimensions: mutual information redundancy and semantic value information. Mutual information redundancy evaluates the data duplication in the tag sequence, and semantic value information evaluates the contribution of each tag information to the overall semantics. The tag sequence is merged from the two dimensions of mutual information redundancy and semantic value information, which reduces the amount of tag information while ensuring the quality of model calculation.

[0035] Furthermore, the acceleration network can be implemented as a plug-and-play module, which is compatible with various understanding and generative multimodal large models. It is easy to use, does not require retraining or complex parameter adjustments, simplifies deployment, greatly shortens the time from R&D to application, and improves the acceleration effect.

[0036] The information-driven model acceleration method from a data perspective provided in the embodiments of the present application can be used in image-to-information search, artificial intelligence generated content (AIGC) model photography, and virtual dressing and fitting to reduce computing costs.

[0037] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0038] Figure 1A flow chart of a model acceleration method provided for an exemplary embodiment of the present application, the method is applied to an acceleration network, the acceleration network includes an acceleration module, and the acceleration module is used to accelerate the neural network model, the acceleration module is located between the first computing module and the second computing module in the neural network model, such as Figure 1 As shown, the method includes:

[0039] 101. Receive a first label sequence output by a first computing module after executing a first model processing operation, where the first label sequence includes a plurality of label information;

[0040] 102. Determine the information degree of at least part of the multiple tag information according to the similarity between the multiple tag information;

[0041] 103. According to the information degree of at least part of the marking information, merge at least part of the marking information to obtain a second marking sequence;

[0042] 104. Input the second tag sequence into a second computing module, so that the second computing module performs a second model processing operation based on the second tag sequence.

[0043] In the present embodiment, the neural network model is a mathematical method that simulates the actual neural network of humans, which is interconnected and exchanges information through a large number of neurons to achieve complex calculation and learning functions. From the perspective of network structure, the neural network model includes but is not limited to: Convolutional Neural Networks (CNN) model, Recurrent Neural Networks (RNN) model, Long Short-Term Memory Network (LSTM) model, Gated Recurrent Units (GRUs), Autoencoder (Autoencoders) model, Generative Adversarial Networks (GAN) model, Variational Autoencoder (VAE) model, Graph Neural Networks (GNN) model and Transformer (Transformer) model, etc. From the perspective of application scenarios, neural network models include: image processing model, natural language processing model, time series analysis model and multimodal model, etc. For example, image processing models can be used for image classification, target detection, semantic segmentation, and image generation; natural language processing models are used for text classification, sentiment analysis, machine translation, and text generation; time series analysis models are used for stock price prediction, weather forecast, and electrocardiogram analysis; multimodal models include multimodal understanding models or multimodal generation models, and please refer to the aforementioned embodiments for details, which will not be repeated here.

[0044] Among them, the multimodal understanding model is a model that can process and fuse multiple modal information (such as vision, text, audio, etc.). This model can handle tasks such as visual question answering, image caption generation, and cross-modal retrieval, aiming to extract meaningful semantic information from data of different modalities and conduct comprehensive understanding.

[0045] Among them, a multimodal generative model is a model that can generate image outputs based on text or image priors, for example, text-to-image generation, or image-to-image generation. "Prior" refers to the known information or background knowledge that the model relies on when generating images. This prior information can be text descriptions, reference images, or other forms of input, which provide guidance and constraints for the model to generate target images.

[0046] For example, for e-commerce applications, tasks such as text-to-product retrieval, image-to-product retrieval, product question and answer, and product labeling can be achieved through multimodal understanding models, while tasks such as text-to-product image generation and editing, fashion model clothing matching, and product image background changing can be achieved through multimodal generation models.

[0047] Among them, the neural network model includes multiple computing modules. The computing module refers to one of the basic components of the neural network model. These computing modules are responsible for performing specific computing operations, usually including but not limited to: linear transformation, nonlinear activation, pooling and normalization. Different computing modules play their own roles in different levels of the neural network and jointly complete the mapping from input to output. The computing modules include but are not limited to: feedforward neural networks (FNN), attention module (ATTention, ATT), fully connected module, convolutional module, activation module, pooling module, normalization module and loss module.

[0048] Among them, the feedforward module (FNNs) can pass the input data through each layer of the feedforward module in sequence, without feedback connection, simple structure, easy to understand and implement. The attention module can process sequence data through the attention mechanism to avoid the sequential calculation bottleneck of RNN. The attention module includes the self-attention module (Self-Attention Module) and the cross-attention module (Cross-Attention Module). Among them, the self-attention module is a mechanism that allows each element in the tag sequence to pay attention to other elements. The cross-attention module is a mechanism that allows elements in one tag sequence (e.g., a visual tag sequence) to pay attention to elements in another tag sequence (e.g., a text tag sequence), which can capture the interaction between different modalities.

[0049] In this embodiment, the acceleration network is used to accelerate the neural network model from a data perspective. For example, the amount of data in the model processing process is reduced by the acceleration module, while the accuracy of the model output can be guaranteed. The number of acceleration modules is not limited. The neural network model can include one acceleration module or multiple acceleration modules, such as 2, 20 or 50. Generally, it can be considered that an acceleration network includes one acceleration module. Of course, an acceleration network can also include multiple acceleration modules, which is not limited.

[0050] The acceleration network can be integrated into the neural network model, or the acceleration network can be plugged into the neural network model as a pluggable module. When the acceleration network is implemented as a pluggable module, there is no need to retrain the neural network model. It can be plug-and-play according to different network structures, which is lightweight, avoids tedious adaptation work, and reduces deployment costs. The acceleration network includes an acceleration module, which can be used to accelerate the neural network model from a data perspective. The acceleration module is located before or after any computing module in the neural network model, or the acceleration module can be located between any two computing modules. For example, the acceleration module is located between the first computing module and the second computing module. Figure 2a shown.

[0051] Among them, the first computing module and the second computing module can be any "connected" computing modules. "Connected" means that when the neural network model does not include an acceleration network, the output of the first computing module can be used as the input of the second computing module. It is explained here that in addition to the output of the first computing module, the second computing module can also include other input information, for example, the output of the first computing module and the input of the first computing module are simultaneously used as the input of the second computing module. Among them, the first computing module and the second computing module can be any connected computing modules in the neural network model. For example, the first computing module is a fully connected module and the second computing module is an attention module; for another example, the first computing module is a residual module and the second computing module is a feedforward module; for another example, the first computing module is a convolution module and the second computing module is a pooling module, etc.

[0052] In this embodiment, a token sequence refers to a series of token information used in the process of processing a task by a neural network model. In the multimodal field, a token sequence can unify data of different modalities, thereby realizing effective processing and fusion of multimodal information. The token sequence includes a plurality of token information, which may include but is not limited to discrete tags or labels such as numerical values, words, subwords, word fragments or characters, etc. The tag information may represent the category label, attribute label or any other form of discrete label of each element in the sequence data. The token sequence can be used as a basic processing unit of a computing module. The token sequence can represent the input data or intermediate processing results of a neural network model. For example, the output of a first computing module may be a token sequence, and the input of a second computing module may also be a token sequence. For ease of distinction and description, the token sequence output by the first computing module performing the first model processing operation is referred to as a first token sequence.

[0053] Among them, the acceleration module can process the labeled sequence, so that the neural network model can handle heterogeneous input features, which is suitable for various visual and language tasks, and enhances the versatility and adaptability of the acceleration module.

[0054] Among them, the first model processing operation is not limited, and it varies depending on the first computing module. For example, if the first computing module is implemented as a conversion module, the first model processing operation can be to convert the input data into a tag sequence; if the first computing module is implemented as a feedforward module, the first model processing operation can be that the input data (such as the tag sequence) is passed through each layer of the feedforward module in sequence without feedback connection; if the first computing module is implemented as a self-attention module, the first model processing operation can be an operation of calculating self-attention for the input tag sequence. If the first computing module is implemented as a cross-attention module, the first model processing operation can be an operation of calculating cross-attention for the input tag sequence. Among them, the input sequence can be two independent tag sequences or two tag sequences spliced ​​together. Among them, whether it is cross-attention or self-attention, it can be calculated by calculating the query (Query), key (Key) and value (Value) involved in the tag sequence.

[0055] In this embodiment, the acceleration module may receive a first tag sequence output by the first computing module, wherein the first tag sequence includes a plurality of tag information. The first tag sequence may include visual tag information alone or text tag information alone, or may include both visual tag information and text tag information. The number of visual tag information is not limited, for example, it may be 200, 500 or 1000, and the number of text tag information is not limited, for example, it may be 4, 10 or 20, and the like.

[0056] In this embodiment, the information degree of at least part of the multiple tag information is determined according to the similarity between the multiple tag information. The at least part of the tag information may be all of the multiple tag information or part of the multiple tag information.

[0057] Among them, the information degree of the tag information is used to measure the importance of the tag information in the first tag sequence. Based on the information degree of the tag information, it is possible to effectively identify which tag information in the first tag sequence is of high importance and which tag information is of low importance. Tag information with high importance can provide more key information, and tag information with low importance provides less key information. Tag information that provides less key information can be merged or deleted, thereby reducing the redundant information of the entire tag sequence and reducing the amount of calculation. At the same time, the performance and output quality of the neural network model can be maintained to the greatest extent. Through this data-level optimization, computing resources are saved from the source, rather than achieving the purpose of acceleration by modifying the model structure. It should be noted that the information degree of the tag information can be positively correlated with the importance of the tag information, for example, the higher the value of the information degree of the tag information, the higher the importance of the tag information. Alternatively, the information degree of the tag information can be negatively correlated with the importance of the tag information, for example, the lower the value of the information degree of the tag information, the higher the importance of the tag information. In the following embodiments, the information degree of the tag information and the importance of the tag information are negatively correlated as an example.

[0058] Optionally, in one implementation, the importance of the tag information can be reflected by the information redundancy between the tag information and other tag information. Based on this, the information degree includes: the mutual information redundancy of the tag information. The mutual information redundancy reflects the similarity between the tag information and other tag information. In other words, it is the proportion of repeated information between the tag information and other tag information. Repeated information does not increase the amount of new information. In short, the mutual information redundancy measures the amount of redundant or predictable information in multiple tag information. The mutual information redundancy helps identify repeated and useless tag information. The higher the mutual information redundancy of the tag information, the higher the similarity between the tag information and other tag information, the more redundant the tag information is or the more it can be predicted or synthesized by other tag information, and the lower the importance of the tag information; the lower the mutual information redundancy of the tag information, the lower the similarity between the tag information and other tag information, the more important the tag information is, and the less likely it is to be replaced.

[0059] Optionally, in another implementation, the importance of the tag information can be reflected not only by the information redundancy between the tag information and other tag information, but also by the amount of useful information contributed by the tag information to the first tag sequence, referred to as semantic value. Based on this, in addition to the mutual information redundancy, the information degree can also include: semantic value information of the tag information. The semantic value information of each tag information indicates the contribution of the tag information in the first tag sequence. The semantic value can ensure that the retained tag information makes an important contribution to the overall task. The higher the semantic value information, the higher the contribution of the tag information, and the lower the semantic value information, the lower the contribution of the tag information.

[0060] For example, the information degree is represented by E, the mutual information redundancy is represented by R, and the semantic value information is represented by A. E is positively correlated with R, and E is negatively correlated with A. The following lists the fusion method of fusing the mutual information redundancy R and the semantic value information A to obtain the information degree E. Among them, the information degree in Table 1 is taken as an example that the importance of the marked information is negatively correlated, that is, the information degree E of the marked information is calculated by the method in Table 1. The lower the information degree E of the marked information, the higher the importance of the marked information, and the more key information can be provided, and the more it is recommended to retain it. The higher the information degree E of the marked information, the lower the importance of the marked information, and the less key information can be provided. It is recommended to delete the marked information or fuse the marked information into other marked information with lower information degrees. As shown in Table 1 below, the fusion method provided in Table 1 is only an example, but is not limited to this.

[0061] Table 1

[0062] Fusion method formula illustrate Linear subtraction E=R-αA Adjust the influence of R and A according to the weight parameter α division E=R / A Balancing the Importance of R and A by Division multiplication E=R(1-αA) Multiplication combines the semantic value of the reduction part Dynamic scaling E=R-β·|γ-Block|·αA Dynamically adjust the R / A ratio at different network layers

[0063] Among them, the weight parameter α is used to balance the mutual information redundancy R and the semantic value information A. α can be a value between 0.1 and 1, β can be 0.1 to 1, and γ can also be 0.1 to 1. α, β and γ can be the same or different. Block represents the network depth of the acceleration module, that is, the number of layers in the neural network model where the acceleration module is located, which determines the complexity of the neural network model and the feature level that can be learned. For example, if the acceleration module includes 5 computing modules before the acceleration module, the network depth Block of the acceleration module is 6. If the acceleration module includes 50 computing modules before the acceleration module, the network depth Block of the acceleration module is 51. The Block value can also be 1, 25 or 80, etc. Among them, "|·|" means taking an absolute value, " / " means division operation, "-" means subtraction operation, and "·" means multiplication operation.

[0064] Among them, the dynamically scaled fusion formula allows flexible adjustment of the ratio of mutual information redundancy and semantic value information between different network layers, further optimizing the performance of the model.

[0065] In general, the lower the mutual information redundancy of the marked information, the higher the semantic value information, the lower the information degree of the marked information, and the more important the marked information; the higher the mutual information redundancy of the marked information, the lower the semantic value information, the higher the information degree of the marked information, and the lower the importance of the marked information.

[0066] Optionally, in another implementation, the importance of the tag information can also be reflected only by the semantic value of the tag information. Based on this, the information degree only includes the semantic value information of the tag information, wherein the method of calculating the semantic value information of the tag information can refer to the above, and in this optional embodiment, the semantic value information of the tag information can be used as the information degree of the tag information.

[0067] In this embodiment, at least part of the tag information is merged according to the information degree of at least part of the tag information to obtain a second tag sequence. For the convenience of distinction and description, the tag sequence output by the second computing module is referred to as the second tag sequence. Among them, the number of tag information included in the second tag sequence is less than the number of at least part of the tag information, and the second tag sequence effectively reduces the number of tag information relative to the first tag sequence, reduces the computational complexity, and maintains or only slightly reduces the performance of the neural network model. In this embodiment, the second tag sequence is input into the second computing module so that the second computing module performs a second model processing operation based on the second tag sequence. The second model processing operation is not limited, depending on the function of the second computing module. For example, if the second computing module is implemented as a feedforward module, the second model processing operation can be the input data (such as the tag sequence) is sequentially transmitted through each layer of the feedforward module without feedback connection; if the second computing module is implemented as a self-attention module, the second model processing operation can be the operation of calculating the self-attention for the input tag sequence. If the second computing module is implemented as a cross-attention module, the second model processing operation can be the operation of calculating the cross-attention for the input tag sequence.

[0068] In an embodiment of the present application, an acceleration network driven by information volume from a data perspective is provided to accelerate a neural network model from the perspective of removing redundant data. The acceleration network uses an acceleration module to accelerate the neural network model. The acceleration module is located between a first computing module and a second computing module in the neural network model. For a first label sequence output by the first computing module, multiple label information is merged according to the information degree of the multiple label information included therein, thereby reducing the amount of calculation involved in subsequent model processing and reducing the calculation cost.

[0069] Furthermore, in an optional embodiment, the information degree includes two dimensions: mutual information redundancy and semantic value information. Mutual information redundancy evaluates data duplication in the tag sequence, and semantic value information evaluates the contribution of each tag information to the overall semantics. The tag sequence is merged from the two dimensions of mutual information redundancy and semantic value information, while reducing the amount of tag information, the quality of model calculation is guaranteed.

[0070] In an optional embodiment, the neural network model may include a multimodal network, wherein the multimodal network is a network that can simultaneously process and fuse input data of multiple different modalities (such as text, images, audio or video, etc.), and improve task performance by integrating information of different modalities. The input data may be a first tag sequence, and the multiple tag information included in the first tag sequence may be divided into two categories, one is tag information of at least two first modalities, and the other is tag information of at least one second modality. Among them, the first modality may be a visual modality, such as an image or video, and the tag information of the first modality is visual tag information; the second modality may be a text modality, for example, text or audio, and the tag information of the second modality is text tag information. Alternatively, the first modality may be a text modality, and the second modality may be a visual modality, without limitation.

[0071] The information degree can be determined for the tag information of the first modality, or for the tag information of the second modality, or for both the tag information of the first modality and the tag information of the second modality. The information degree of the tag information of the first modality can be determined with the assistance of the tag information of the second modality, or the tag information of the first modality can be determined with the assistance of the tag information of the second modality. The specific auxiliary determination process can refer to the auxiliary determination of the calculation of semantic value information, which can be specifically referred to in subsequent embodiments and will not be described in detail here.

[0072] For example, for the tag information of modality A1 (e.g., visual modality), it may be impossible to determine which tag information has high semantic value information and which tag information has low semantic value information. Therefore, the tag information of modality A2 (e.g., text modality) can be used to assist in determining which tag information of modality A1 has high semantic value information and which has low semantic value information.

[0073] The implementation method of determining the information degree of at least part of the multiple tag information according to the similarity between the multiple tag information is not limited. The following takes the determination of the information degree for the tag information of the first modality as an example to introduce the implementation method of determining the information degree of at least part of the multiple tag information according to the similarity between the multiple tag information.

[0074] Specifically, based on the similarity between the label information of at least two first modalities, the mutual information redundancy of the label information of at least two first modalities is generated; based on the similarity between the label information of at least two first modalities and the label information of at least one second modality, the semantic value information of the label information of at least two first modalities is generated; based on the mutual information redundancy and the semantic value information of the label information of at least two first modalities, the information degree of the label information of at least two first modalities is determined.

[0075] Among them, the specific implementation method of determining the information degree of the labeled information of at least two first modalities according to the mutual information redundancy and semantic value information of the labeled information of at least two first modalities can be seen in Table 1, and the fusion method can be randomly selected from Table 1 to determine the information of the labeled information of at least two first modalities. Alternatively, when the network depth of the neural network model is less than the set depth threshold, the fusion method is randomly selected from the three methods of linear subtraction, division and multiplication in Table 1 to determine the information of the labeled information of at least two first modalities. When the network depth of the neural network model is greater than or equal to the set depth threshold, the dynamic scaling method in Table 1 is used to determine the information of the labeled information of at least two first modalities.

[0076] Optionally, the implementation method of generating the mutual information redundancy of the label information of at least two first modalities according to the similarity between the label information of at least two first modalities is not limited, and a specific implementation method is provided below.

[0077] For example, the similarity between the tag information of any first modality (hereinafter referred to as any tag information) and the tag information of other first modalities is calculated to obtain at least one candidate similarity corresponding to the any tag information, and a candidate similarity greater than a set similarity threshold is selected as the mutual information redundancy of the any tag information, and the similarity threshold may be 80%, 90% or 95%, etc. Further, if there are multiple candidate similarities greater than the set similarity threshold, one candidate similarity may be randomly selected from the multiple candidate similarities as the mutual information redundancy of the any tag information.

[0078] For another example, the similarity between the label information of any first modality and the label information of other first modalities is calculated to obtain at least one candidate similarity, and the largest candidate similarity is selected as the mutual information redundancy of the label information of any first modality.

[0079] Among them, calculating the similarity between the tag information of each first modality and the tag information of other first modalities can specifically be calculating the cosine similarity between the tag information of each first modality and the tag information of other first modalities. Optionally, the implementation method of generating the semantic value information of at least two first modal tag information based on the similarity between the tag information of at least two first modalities and the tag information of at least one second modality is not limited. The following provides a specific implementation method, in the case where there are at least two second modal tag information, the tag information of at least two second modalities is fused to obtain the target tag information, such as Figure 3b As shown; calculating the similarity between the tag information of at least two first modalities and the target tag information respectively; generating semantic value information of the tag information of at least two first modalities according to the similarity between the tag information of at least two first modalities and the target tag information respectively.

[0080] The implementation method of fusing the tag information of at least two second modalities to obtain the target tag information is not limited. For example, the tag information of at least two second modalities is weighted averaged to obtain the target tag information. For another example, the tag information of at least two second modalities can be input into a self-attention module, and the tag information of at least two second modalities is fused through the self-attention module.

[0081] In another optional embodiment, the neural network model may include a unimodal network, which is a network that processes a single type of input data. In the case where the first computing module belongs to a unimodal network, the implementation method of determining the information degree of at least part of the multiple tag information based on the similarity between the multiple tag information is not limited. Considering that the unimodal network includes a tag sequence of one modality, in this case, semantic value information may not be considered. A specific implementation method is provided below, which calculates the mutual information redundancy of multiple tag information based on the similarity between the multiple tag information; and determines the information degree of multiple tag information based on the mutual information redundancy of the multiple tag information.

[0082] Among them, the implementation method of calculating the mutual information redundancy of multiple label information based on the similarity between multiple label information is not limited. For example, the similarity between any label information and other label information is calculated to obtain multiple candidate similarities corresponding to the any label information, and the largest candidate similarity is selected from the multiple candidate similarities as the mutual information redundancy of the any label information. For another example, the similarity between any label information and other label information is calculated to obtain multiple candidate similarities corresponding to the any label information, and the candidate similarity greater than the set similarity threshold is selected from the multiple candidate similarities as the mutual information redundancy of the any label information. If there are multiple candidate similarities greater than the set similarity threshold, one is randomly selected from them as the mutual information redundancy of the any label information.

[0083] It should be noted that, whether it is a unimodal network or a multimodal network, when generating the mutual information redundancy of the labeled information, the efficiency of generating the mutual information redundancy of the labeled information can be improved by grouping the labeled information.

[0084] The mutual information redundancy of generating multiple labeled information is taken as an example, but it is not limited to this. The multiple labeled information is input into the attention module, and the multiple labeled information is divided into two groups by the attention module, namely group P and group Q. If the number of the multiple labeled information is an even number, the number of labeled information included in group P and group Q is the same. If the number of the multiple labeled information is an odd number, the number of labeled information included in group P is greater than the number of labeled information included in group Q, or the number of labeled information included in group Q is greater than the number of labeled information included in group P.

[0085] For example, group P includes label information p1 to label information p4, and group Q includes label information q1 to label information q5. For label information p1 in group P, its similarity with label information q1 to label information q5 is calculated respectively, and 5 candidate similarities are obtained. The largest candidate similarity is selected as the mutual information redundancy of label information p1. Similarly, for label information p2 to label information p4, the mutual information redundancy can be determined based on the same method as label information p1. For label information q1 in group Q, its similarity with label information p1 to label information p4 is calculated respectively, and 4 candidate similarities are obtained. The largest candidate similarity is selected as the mutual information redundancy of label information q1. Similarly, for label information q2 to label information q5, the mutual information redundancy can be determined based on the same method as label information q1.

[0086] In an optional embodiment, the implementation method of combining at least part of the tag information according to the information degree of at least part of the tag information to obtain the second tag sequence is not limited. The following provides an example, according to the information degree of at least part of the tag information, the first tag information is selected from it; according to the first tag information, at least part of the tag information is combined to obtain the second tag sequence. Figure 3b In the figure, the first tag sequence, the second tag sequence and the third tag sequence are exemplarily displayed from the dimensions of visual tag sequence and text tag sequence respectively.

[0087] Optionally, the implementation method of selecting the first marking information according to the information degree of at least part of the marking information is not limited, and is exemplarily described below.

[0088] Example D1, according to the information degree of at least part of the label information, the first label information below the information degree threshold is selected, and the information degree threshold may be 0.1, 0.2 or 0.3, etc.

[0089] Example D2, in the order of the information degree of at least part of the marking information from small to large, select a set number of marking information as the first marking information. Wherein, the number of at least part of the marking information is N, and the set number can be N / 2, 2N / 3 or 4N / 5, etc. For example, the information degree of at least part of the marking information is traversed, and the marking information with the smallest information degree is selected from it each time. After multiple traversals, until a set number of marking information is obtained, the set number of marking information is used as the first marking information. For another example, at least part of the marking information can be sorted in the order of the information degree of at least part of the marking information from high to low, and a set number of marking information can be selected from the sorted at least part of the marking information as the first marking information; or at least part of the marking information can be sorted in the order of the information degree of at least part of the marking information from low to high, so as to select a set number of marking information from the sorted at least part of the marking information as the first marking information.

[0090] Optionally, according to the first tag information, at least part of the tag information is combined to obtain the second tag sequence. The implementation manner is not limited. An exemplary description is given below.

[0091] Example F1, taking the first tag information as a merging result of at least part of the tag information; wherein the amount of tag information included in the merging result is less than the amount of at least part of the tag information; and generating a second tag sequence according to the merging result.

[0092] Example F2: For the convenience of distinction and description, the tag information other than the first tag information in at least part of the tag information is referred to as the second tag information, and the second tag information is fused into the first tag information as the merging result of at least part of the tag information. For example, an attention mechanism is used to fuse the second tag information into the first tag information to obtain the fused first tag information, and the fused first tag information is used as the merging result of at least part of the tag information; and a second tag sequence is generated according to the merging result.

[0093] Among them, Figure 3b In FIG. 1 , the solid line portion in the second marking sequence represents the first marking information, and the dotted line portion represents the second marking information.

[0094] Among them, for the convenience of distinction and description, the tag information whose information degree is not calculated in the multiple tag information is referred to as other tag information. For example, the other tag information can be the tag information of the aforementioned second mode. Whether it is the above-mentioned example F1 or example F2, when at least part of the tag information is implemented as part of the tag information in the multiple tag information, a second tag sequence is generated together according to the merge result and other tag information. For example, the tag information included in the merge result and the other tag information are used as the tag information in the second tag sequence. When at least part of the tag information is implemented as all of the multiple tag information, the tag information included in the merge result can be directly used as the tag information included in the second tag sequence.

[0095] In an optional embodiment, the neural network model is implemented as a multimodal model, which includes multiple unimodal networks and a multimodal fusion network. The multimodal fusion network is used to fuse the output results of multiple unimodal networks. The multiple unimodal networks may include but are not limited to: text unimodal networks, image unimodal networks, video unimodal networks, and audio unimodal networks, etc.

[0096] The relationship between the first calculation module and the second calculation module and the multimodal model is not limited, and an exemplary description is given below.

[0097] Example G1, the first computing module and the second computing module are distributed in the same unimodal network, for example, the first computing module and the second computing module are both distributed in an image unimodal network or a text unimodal network.

[0098] Example G2, the first computing module and the second computing module are distributed in a multimodal fusion network.

[0099] Example G3, the first computing module and the second computing module are distributed in a unimodal network and a multimodal fusion network, respectively. For example, the first computing module is distributed in an image unimodal network, and the second computing module is distributed in a multimodal fusion network. For another example, the first computing module is distributed in a text unimodal network, and the second computing module is distributed in a multimodal fusion network.

[0100] Optionally, when the first computing module and the second computing module are distributed in the same unimodal network, the first computing module is a feedforward module, and the second computing module is a self-attention module or a feedforward module; or, the first computing module is a conversion module, and the second computing module is a feedforward module or a self-attention module.

[0101] When the first computing module and the second computing module are distributed in the multimodal fusion network, the first computing module is a feedforward module, and the second computing module is a cross-attention module or a feedforward module; the second computing module is a cross-attention module, and the first computing module is a cross-attention module or a feedforward module.

[0102] like Figure 3a As shown, the multimodal model is implemented as a multimodal generation model and a multimodal understanding model as an example to illustrate the first computing module and the second computing module, but it is not limited to this. In the multimodal understanding model, the first computing module is an attention module and the second computing module is a feedforward module as an example, but it is not limited to this. In the multimodal generation model, the second computing module is an attention network as an example, and the first computing module is not shown.

[0103] In an optional embodiment, when the neural network model is implemented as a multimodal model, and the multimodal model is a multimodal generative model, the acceleration network further includes a restoration module for restoring the tag sequence to ensure that the length of the tag sequence remains constant and the accuracy of subsequent processing is guaranteed. Usually, one acceleration module corresponds to a restoration module. Figure 2b As shown, the restoration module is located after the second calculation module as an example, but it is not limited to this. Figure 2b The first computing module is not shown.

[0104] The method also includes: inputting the third marker sequence output by the second calculation module performing the second model processing operation into the restoration module, and performing restoration processing on the third marker sequence in the restoration module to obtain a fourth marker sequence; wherein the number of marker information included in the fourth marker sequence is adapted to the number of marker information included in the first marker sequence, for example, the number of marker information included in the fourth marker sequence is the same as the data of the marker information included in the first marker sequence, or the number of marker information included in the fourth marker sequence is one more or one less than the number of marker information included in the first marker sequence, and there is no limitation on this.

[0105] Optionally, in the process of merging at least part of the tag information, the merge description information is recorded, and the merge description information is used to describe the mapping relationship between the merging result of at least part of the tag information and at least part of the tag information. Among them, an implementation method of restoring the third tag sequence to obtain a fourth tag sequence includes: according to the mapping relationship included in the merge description information, the third tag information in the third tag sequence is reversely mapped to obtain the fourth tag sequence. For example, the merge description information may be a mapping relationship between the second tag information and the first tag information. In the restoration module, according to the mapping relationship between the second tag information and the first tag information, the third tag information in the third tag sequence is reversely mapped to obtain the fourth tag sequence.

[0106] The model acceleration method provided in the embodiment of the present application has the following advantages:

[0107] (1) Explore reasoning acceleration from the data level: Combine mutual information redundancy and semantic value information in information calculation to ensure that not only redundant information between labeled information is considered, but also important semantic information is retained. This method removes redundancy from a data perspective and achieves a balance between efficiency and performance.

[0108] (2) Applicable to multimodal tasks: Model acceleration methods are applicable to a variety of comprehension and generation of multimodal large models, and can handle multimodal tasks (such as cross-modal retrieval, visual question answering, image description, etc.) and unimodal tasks (such as image classification), showing strong generalization capabilities. Specifically, the accelerated network can achieve significant performance improvements in image-text comprehension and text-image generation tasks without the need for special optimization for each task.

[0109] Furthermore, visual tag information and text tag information can be processed, and the optimization of multimodal tasks can be achieved through the calculation of cross-modal semantic value information, making model acceleration have a wider application prospect in multimodal tasks.

[0110] (3) Strong universality: When the acceleration network is implemented as a plug-in module, it can be easily integrated into existing pre-trained large models (neural network models), including unimodal visual models, multimodal understanding models, and multimodal generation models, without the need to retrain the model, significantly reducing the complexity and cost of use and improving the adaptability and scalability of the model.

[0111] (4) The multimodal understanding model can efficiently process the labeled sequence and maintain the model performance: By introducing the information degree calculation in the acceleration module, the amount of labeled information can be effectively reduced, the computational complexity can be reduced, and the performance of the multimodal understanding model can be maintained or only slightly reduced.

[0112] (5) Improving the generation efficiency of multimodal generative models and accelerating reasoning: The information-driven acceleration network introduces an acceleration module between the first computing module and the second computing module, and introduces a restoration module after the second computing module, which greatly reduces the repeated calculation of redundant information and improves the reasoning speed and computing efficiency of the multimodal generative model.

[0113] In addition, the accelerated network can be used not only in the reasoning process of the neural network model, but also in the training process of the neural network model.

[0114] (6) Plug and play: Simply insert the information-driven acceleration network into the existing neural network model without retraining or complex parameter adjustments. This not only simplifies the deployment process, but also greatly shortens the time from R&D to application. For systems that are already using other model acceleration methods (such as pruning and quantization), the information-driven acceleration network can be directly superimposed to further improve the acceleration effect, provide higher computing efficiency and better user experience.

[0115] In addition to providing the above-mentioned model acceleration method, the embodiment of the present application also provides a data processing method, which includes:

[0116] Responding to an access request of a target service and obtaining service demand information, the target service corresponds to a neural network model, and the neural network model includes at least a first computing module, a second computing module, and an acceleration network;

[0117] According to the service demand information, the neural network model is called to perform model reasoning operations on the target service, and the acceleration network accelerates the neural network model during the execution of the model reasoning operations to obtain service result information of the target service;

[0118] The service result information is returned to the requester of the service request so that the requester can use the target service.

[0119] The target service may be any service that uses a neural network model. For example, the target service may include, but is not limited to, classification services, natural language processing services, image generation services, and intelligent question-and-answer services. Furthermore, in the field of e-commerce, the target service may include, but is not limited to, services such as text-to-product retrieval, image-to-product retrieval, product question-and-answer, product labeling, and services such as text-to-product image generation and editing, fashion model clothing matching, and product image background change.

[0120] In this embodiment, the service demand information may include, but is not limited to: images, text, videos, and audio, etc. Among them, the service demand information can be input into a neural network model, and the service demand information can be converted into a tag sequence in the neural network model. For example, if the service demand information is implemented as an image, the image can be divided into multiple image blocks, and each image block can be converted into a visual tag information, thereby obtaining a visual tag sequence. If the service demand information is implemented as text, the text is cut into single characters, single times or fragments, etc., and mapped into multiple text tag information corresponding to the text, thereby obtaining a text tag sequence. If the service demand information is implemented as a video, the video can be split into multiple image frames, and the multiple image frames can be converted into a tag sequence, or multiple video frames can be fused into a target video frame, and a visual tag sequence is generated for the target video frame. If the service demand information is implemented as audio, a text tag sequence can be obtained in a similar manner to text.

[0121] In this embodiment, the service result information may be image information, text information, video information or audio information. When the neural network model is implemented as a multimodal model, the service demand information may include information of multiple modalities, and the service result information may include information of at least one modality. For example, if the service demand information is text information, the service result information may be image information; if the service demand information is text information and first image information, the service result information may be second image information.

[0122] In this embodiment, other similar introductions can refer to the above-mentioned embodiment, which will not be repeated here. This embodiment of the application also provides another data processing method, which includes:

[0123] 301. In response to a service request for a target service, obtain service demand information. The target service corresponds to a multimodal generation model. The multimodal generation model includes at least a feedforward module, an attention module, and an acceleration network. The acceleration network includes an acceleration module located before the target module and a restoration module located after the target module. The target module is any one of the feedforward module and the attention module.

[0124] 302. According to the service demand information, the multimodal generation model is called to perform a model reasoning operation on the target service. The acceleration module is used to accelerate the multimodal generation model during the execution of the model reasoning operation. The restoration module is used to restore the result of the model acceleration to obtain the service result information of the target service.

[0125] 303. Return the service result information to the requesting end of the service request so that the requesting end can provide the target service.

[0126] In this embodiment, the target module may correspond to the second calculation module in the above embodiment. The detailed description of the data processing method can be found in the above embodiment, which will not be repeated here.

[0127] Regarding the embodiments of the present application Figure 1 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0128] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0129] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0130] Figure 4 A schematic diagram of a model acceleration device provided by an exemplary embodiment of the present application, wherein the device corresponds to an acceleration network, wherein the acceleration network uses an acceleration module to accelerate a neural network model, wherein the acceleration module is located between a first computing module and a second computing module in the neural network model, such as Figure 4 As shown, the device includes: a receiving module 41, a determining module 42, a merging module 43 and an input module 44.

[0131] A receiving module 41 is used to receive a first label sequence output by the first computing module after executing the first model processing operation, where the first label sequence includes a plurality of label information;

[0132] A determination module 42, configured to determine the information degree of at least part of the plurality of marking information according to the similarity between the plurality of marking information;

[0133] A merging module 43, configured to merge at least part of the marking information according to the information degree of at least part of the marking information to obtain a second marking sequence;

[0134] The input module 44 is used to input the second label sequence into the second calculation module, so that the second calculation module performs a second model processing operation based on the second label sequence.

[0135] In an optional embodiment, when the first computing module belongs to a multimodal fusion network, the multiple labeling information includes: labeling information of at least two first modalities and labeling information of at least one second modality; the determination module is specifically used to: generate mutual information redundancy of at least two labeling information of the first modalities according to the similarity between the labeling information of the at least two first modalities; generate semantic value information of the labeling information of the at least two first modalities according to the similarity between the labeling information of the at least two first modalities and the labeling information of the at least one second modality; determine the information degree of the labeling information of the at least two first modalities according to the mutual information redundancy and semantic value information of the labeling information of the at least two first modalities.

[0136] Optionally, the determination module is specifically used to: when there are at least two pieces of tag information of the second modality, fuse the at least two pieces of tag information of the second modalities to obtain target tag information; calculate the similarity between the at least two pieces of tag information of the first modalities and the target tag information; and generate semantic value information of the at least two pieces of tag information of the first modalities based on the similarity between the at least two pieces of tag information of the first modalities and the target tag information.

[0137] Optionally, the marking information of the first modality is visual marking information, and the marking information of the second modality is textual marking information.

[0138] In an optional embodiment, when the first calculation module belongs to a unimodal network, the determination module is specifically used to: calculate the mutual information redundancy of multiple label information according to the similarity between the multiple label information; and determine the information degree of the multiple label information according to the mutual information redundancy of the multiple label information.

[0139] In an optional embodiment, the merging module is specifically used to: select first tag information from at least part of the tag information according to the information degree of at least part of the tag information; and merge at least part of the tag information according to the first tag information to obtain a second tag sequence.

[0140] Optionally, the merging module is specifically used to: use the first tag information as the merging result of at least part of the tag information, or merge the second tag information into the first tag information as the merging result of at least part of the tag information; generate a second tag sequence according to the merging result, wherein the second tag information is the tag information in at least part of the tag information except the first tag information.

[0141] In an optional embodiment, the neural network model is a multimodal model, which includes multiple unimodal networks and a multimodal fusion network, and the multimodal fusion network is used to fuse the output results of multiple unimodal networks; wherein the first computing module and the second computing module are distributed in the same unimodal network; or, the first computing module and the second computing module are distributed in the multimodal fusion network; or, the first computing module and the second computing module are respectively distributed in the unimodal network and the multimodal fusion network.

[0142] Optionally, when the first computing module and the second computing module are distributed in the same unimodal network, the first computing module is a feedforward module, and the second computing module is a self-attention module or a feedforward module; or, the first computing module is a conversion module, and the second computing module is a feedforward module or a self-attention module; when the first computing module and the second computing module are distributed in a multimodal fusion network, the first computing module is a feedforward module, and the second computing module is a cross-attention module or a feedforward module; the second computing module is a cross-attention module, and the first computing module is a cross-attention module or a feedforward module.

[0143] In an optional embodiment, when the neural network model is a multimodal model and the multimodal model is a multimodal generative model, the acceleration network also includes a restoration module; the input module is also used to: input the third label sequence output by the second computing module performing the second model processing operation into the restoration module, and the restoration module is used to restore the third label sequence to obtain a fourth label sequence; wherein the number of label information included in the fourth label sequence is adapted to the number of label information included in the first label sequence.

[0144] Optionally, the device also includes a recording module; the recording module is used to record merge description information during the process of merging at least part of the tag information, and the merge description information is used to describe the mapping relationship between the merging result of at least part of the tag information and at least part of the tag information; the restoration mold body is used to: according to the mapping relationship included in the merge description information, reversely map the third tag information in the third tag sequence to obtain a fourth tag sequence.

[0145] In an optional embodiment, the acceleration network is integrated into the neural network model, or the acceleration network is plugged into the neural network model as a pluggable module.

[0146] The embodiment of the present application also provides a data processing device, which includes: an acquisition module, a calling module and a return module;

[0147] The acquisition module is used to respond to the access request of the target service and obtain the service demand information. The target service corresponds to a neural network model;

[0148] A calling module is used to call the neural network model to perform model reasoning operations on the target service according to the service demand information, so as to obtain the service result information of the target service;

[0149] A return module is used to return the service result information to the requesting end of the service request so that the requesting end can use the target service; wherein the neural network model includes at least a first computing module, a second computing module and an acceleration network; wherein the acceleration network uses the model acceleration method provided in an embodiment of the present application to accelerate the neural network model.

[0150] The embodiment of the present application also provides a data processing method, the device comprising: an acquisition module, a calling module and a return module;

[0151] An acquisition module, for obtaining service demand information in response to a service request for a target service, wherein the target service has a multimodal generation model;

[0152] A calling module is used to call the multimodal generation model to perform model reasoning operations on the target service according to the service demand information, so as to obtain service result information of the target service;

[0153] A return module is used to return the service result information to the requesting end of the service request so that the requesting end can provide the target service; wherein the multimodal generation model includes at least a feedforward module, an attention module and an acceleration network, the acceleration network includes an acceleration module located before the target module and a restoration module located after the target module, and the target module is any module among the feedforward module and the attention module; wherein the acceleration network adopts the model acceleration method provided in the embodiment of the present application to perform model acceleration for the multimodal generation model.

[0154] Regarding the embodiments of the present application Figure 4 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0155] Figure 5 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present application corresponds to an acceleration network, wherein the acceleration network uses an acceleration module to accelerate a neural network model, and the acceleration module is located between a first computing module and a second computing module in the neural network model, such as Figure 5 As shown, the device includes: a memory 54 and a processor 55.

[0156] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device.

[0157] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54, so as to: receive a first label sequence output by the first computing module when executing the first model processing operation, the first label sequence including multiple label information; determine the information degree of at least part of the multiple label information according to the similarity between the multiple label information; merge at least part of the label information according to the information degree of at least part of the label information to obtain a second label sequence; input the second label sequence into the second computing module, so that the second computing module performs the second model processing operation based on the second label sequence.

[0158] In an optional embodiment, when the first computing module belongs to a multimodal fusion network, the multiple labeling information includes: labeling information of at least two first modalities and labeling information of at least one second modality; when the processor 55 determines the information degree of at least part of the multiple labeling information according to the similarity between the multiple labeling information, it is specifically used to: generate mutual information redundancy of at least two labeling information of the first modalities according to the similarity between the labeling information of the at least two first modalities; generate semantic value information of at least two labeling information of the first modalities according to the similarity between the labeling information of the at least two first modalities and the labeling information of at least one second modality; determine the information degree of at least two labeling information of the first modalities according to the mutual information redundancy and semantic value information of the labeling information of the at least two first modalities.

[0159] Optionally, when the processor 55 generates semantic value information of at least two pieces of tag information of the first modalities based on the similarity between the tag information of the at least two first modalities and the tag information of the at least one second modalities, the processor 55 is specifically used to: when there are at least two pieces of tag information of the second modalities, fuse the tag information of the at least two second modalities to obtain target tag information; calculate the similarity between the tag information of the at least two first modalities and the target tag information respectively; and generate semantic value information of the tag information of the at least two first modalities based on the similarity between the tag information of the at least two first modalities and the target tag information respectively.

[0160] Optionally, the marking information of the first modality is visual marking information, and the marking information of the second modality is textual marking information.

[0161] In an optional embodiment, when the processor 55 determines the information degree of at least part of the multiple tag information based on the similarity between the multiple tag information when the first computing module belongs to a unimodal network, the processor is specifically used to: calculate the mutual information redundancy of the multiple tag information based on the similarity between the multiple tag information; determine the information degree of the multiple tag information based on the mutual information redundancy of the multiple tag information.

[0162] In an optional embodiment, when the processor 55 merges at least part of the tag information according to the information degree of at least part of the tag information to obtain the second tag sequence, it is specifically used to: select the first tag information from the at least part of the tag information according to the information degree of at least part of the tag information; merge at least part of the tag information according to the first tag information to obtain the second tag sequence.

[0163] Optionally, when the processor 55 merges at least part of the tag information according to the first tag information to obtain a second tag sequence, it is specifically used to: use the first tag information as the merging result of at least part of the tag information, or merge the second tag information into the first tag information as the merging result of at least part of the tag information; generate a second tag sequence according to the merging result, wherein the second tag information is the tag information in at least part of the tag information except the first tag information.

[0164] In an optional embodiment, the neural network model is a multimodal model, which includes multiple unimodal networks and a multimodal fusion network, and the multimodal fusion network is used to fuse the output results of multiple unimodal networks; wherein the first computing module and the second computing module are distributed in the same unimodal network; or, the first computing module and the second computing module are distributed in the multimodal fusion network; or, the first computing module and the second computing module are respectively distributed in the unimodal network and the multimodal fusion network.

[0165] Optionally, when the first computing module and the second computing module are distributed in the same unimodal network, the first computing module is a feedforward module, and the second computing module is a self-attention module or a feedforward module; or, the first computing module is a conversion module, and the second computing module is a feedforward module or a self-attention module; when the first computing module and the second computing module are distributed in a multimodal fusion network, the first computing module is a feedforward module, and the second computing module is a cross-attention module or a feedforward module; the second computing module is a cross-attention module, and the first computing module is a cross-attention module or a feedforward module.

[0166] In an optional embodiment, when the neural network model is a multimodal model and the multimodal model is a multimodal generative model, the acceleration network also includes a restoration module, and the processor 55 is further used to: input the third label sequence output by the second computing module when performing the second model processing operation into the restoration module, and restore the third label sequence to obtain a fourth label sequence; wherein the number of label information included in the fourth label sequence is adapted to the number of label information included in the first label sequence.

[0167] Optionally, the processor 55 is also used to: record merge description information during the process of merging at least part of the tag information, where the merge description information is used to describe the mapping relationship between the merging result of at least part of the tag information and at least part of the tag information; when the processor 55 restores the third tag sequence to obtain a fourth tag sequence, it is specifically used to: reverse map the third tag information in the third tag sequence according to the mapping relationship included in the merge description information to obtain a fourth tag sequence.

[0168] In an optional embodiment, the acceleration network is integrated into the neural network model, or the acceleration network is plugged into the neural network model as a pluggable module.

[0169] Regarding the embodiments of the present application Figure 5 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0170] Further, if Figure 5 As shown, the electronic device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT (Internet of Things) device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.

[0171] The embodiment of the present application also provides an electronic device, the implementation structure of the electronic device is Figure 5 The implementation structure of the electronic device shown is the same or similar, and can refer to Figure 5 The electronic device provided in this embodiment is implemented as follows: Figure 5 The difference between the electronic devices in the illustrated embodiments mainly lies in that the functions implemented by the processor executing the computer program stored in the memory are different. For the electronic device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: respond to the access request of the target service and obtain the service demand information, and the target service corresponds to a neural network model; according to the service demand information, call the neural network model to perform model reasoning operations on the target service to obtain the service result information of the target service; return the service result information to the request end of the service request, so that the request end can use the target service; wherein the neural network model includes at least a first computing module, a second computing module and an acceleration network; wherein the acceleration network uses the model acceleration method provided in the embodiment of the present application to accelerate the neural network model.

[0172] The embodiment of the present application also provides an electronic device, the implementation structure of the electronic device is Figure 5 The implementation structure of the electronic device shown is the same or similar, and can refer to Figure 5 The electronic device provided in this embodiment is implemented as follows: Figure 5 The difference between the electronic devices in the illustrated embodiments mainly lies in that the functions implemented by the processor executing the computer program stored in the memory are different. For the electronic device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: respond to the service request for the target service, obtain service demand information, and the target service corresponds to a multimodal generation model; according to the service demand information, call the multimodal generation model to perform model reasoning operations on the target service to obtain the service result information of the target service; return the service result information to the request end of the service request, so that the request end can provide the target service; wherein, the multimodal generation model at least includes a feedforward module, an attention module and an acceleration network, and the acceleration network includes an acceleration module located before the target module and a restoration module located after the target module, and the target module is any module of the feedforward module and the attention module; wherein, the acceleration network adopts the model acceleration method provided in the embodiment of the present application to perform model acceleration for the multimodal generation model.

[0173] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0174] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed, can implement the various steps that can be executed by an electronic device in the above-mentioned model acceleration method and data processing method embodiments.

[0175] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the processor implements the steps that can be executed by an electronic device in the above-mentioned model acceleration method and data processing method embodiments.

[0176] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0177] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0178] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0179] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0180] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0182] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0185] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0186] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0187] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0188] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0189] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A model acceleration method, characterized in that: Applied to an acceleration network, the acceleration network uses an acceleration module to accelerate a neural network model, the acceleration module is located between a first computing module and a second computing module in the neural network model, and the method includes: Receive a first label sequence output by the first computing module after executing a first model processing operation, wherein the first label sequence includes a plurality of label information; Determining the information degree of at least part of the plurality of tag information according to the similarities between the plurality of tag information; According to the information degree of at least part of the tag information, the at least part of the tag information is merged to obtain a second tag sequence; The second label sequence is input into the second computing module so that the second computing module performs a second model processing operation based on the second label sequence.

2. The method according to claim 1, characterized in that In the case where the first computing module belongs to a multimodal fusion network, the plurality of label information includes: label information of at least two first modalities and label information of at least one second modality; Determining the information degree of at least part of the plurality of tag information according to the similarities between the plurality of tag information includes: Generate mutual information redundancy of the label information of the at least two first modalities according to the similarity between the label information of the at least two first modalities; generating semantic value information of the at least two first modal marking information according to the similarity between the at least two first modal marking information and the at least one second modal marking information; The information degree of the label information of the at least two first modalities is determined according to the mutual information redundancy and the semantic value information of the label information of the at least two first modalities.

3. The method according to claim 2, characterized in that Generating semantic value information of the at least two first modal marking information according to the similarity between the at least two first modal marking information and the at least one second modal marking information, comprising: When there are at least two pieces of label information of the second modality, fusing the at least two pieces of label information of the second modality to obtain target label information; Calculating the similarity between the tag information of the at least two first modalities and the target tag information respectively; According to the similarities between the tag information of the at least two first modalities and the target tag information, semantic value information of the tag information of the at least two first modalities is generated.

4. The method according to claim 2, characterized in that: The marking information of the first modality is visual marking information, and the marking information of the second modality is textual marking information.

5. The method according to claim 1, characterized in that In the case where the first calculation module belongs to a unimodal network, determining the information degree of at least part of the plurality of label information according to the similarities between the plurality of label information includes: Calculating mutual information redundancy of the plurality of label information according to similarities between the plurality of label information; The information degree of the plurality of label information is determined according to the mutual information redundancy of the plurality of label information.

6. The method according to claim 1, characterized in that According to the information degree of at least part of the tag information, the at least part of the tag information is merged to obtain a second tag sequence, including: selecting first marking information from the at least part of the marking information according to the information degree of the at least part of the marking information; According to the first marking information, the at least part of the marking information is merged to obtain the second marking sequence.

7. The method according to claim 6, characterized in that According to the first marking information, merging at least part of the marking information to obtain the second marking sequence includes: Using the first tag information as the merging result of the at least part of the tag information, or fusing the second tag information into the first tag information as the merging result of the at least part of the tag information; The second marking sequence is generated according to the merging result, wherein the second marking information is marking information in the at least part of the marking information except the first marking information.

8. The method according to any one of claims 1 to 7, characterized in that: The neural network model is a multimodal model, which includes a plurality of single-modal networks and a multimodal fusion network, and the multimodal fusion network is used to fuse the output results of the plurality of single-modal networks; Wherein, the first computing module and the second computing module are distributed in the same single-mode network; Alternatively, the first computing module and the second computing module are distributed in the multimodal fusion network; Alternatively, the first computing module and the second computing module are distributed in a single-modal network and the multi-modal fusion network, respectively.

9. The method according to claim 8, characterized in that In the case where the first computing module and the second computing module are distributed in the same unimodal network, the first computing module is a feedforward module, and the second computing module is a self-attention module or a feedforward module; or, the first computing module is a conversion module, and the second computing module is a feedforward module or a self-attention module; When the first computing module and the second computing module are distributed in the multimodal fusion network, the first computing module is a feedforward module, and the second computing module is a cross-attention module or a feedforward module; the second computing module is a cross-attention module, and the first computing module is a cross-attention module or a feedforward module.

10. The method according to any one of claims 1 to 7 and 9, characterized in that: When the neural network model is a multimodal model and the multimodal model is a multimodal generation model, the acceleration network further includes a restoration module, and the method further includes: Inputting the third label sequence output by the second computing module after executing the second model processing operation into the restoration module, and performing restoration processing on the third label sequence to obtain a fourth label sequence; The amount of marking information included in the fourth marking sequence is adapted to the amount of marking information included in the first marking sequence.

11. The method according to claim 10, characterized in that Also includes: In the process of merging the at least part of the tag information, recording merging description information, where the merging description information is used to describe a mapping relationship between a merging result of the at least part of the tag information and the at least part of the tag information; Performing a reduction process on the third marker sequence to obtain a fourth marker sequence includes: According to the mapping relationship included in the combined description information, the third tag information in the third tag sequence is reversely mapped to obtain a fourth tag sequence.

12. The method according to any one of claims 1-7, 9, 11, characterized in that: The acceleration network is integrated into the neural network model, or the acceleration network is plugged into the neural network model as a pluggable module.

13. A data processing method, characterized in that: The method comprises: Responding to an access request of a target service and obtaining service demand information, wherein the target service corresponds to a neural network model; According to the service demand information, calling the neural network model to perform a model reasoning operation on the target service to obtain service result information of the target service; Returning the service result information to the requesting end of the service request so that the requesting end can use the target service; Wherein, the neural network model includes at least a first computing module, a second computing module and an acceleration network; wherein, the acceleration network adopts the method described in any one of claims 1-12 to accelerate the model of the neural network model.

14. A data processing method, characterized in that: The method comprises: Responding to a service request for a target service, obtaining service demand information, the target service corresponding to a multimodal generation model; According to the service demand information, calling the multimodal generation model to perform a model reasoning operation on the target service to obtain service result information of the target service; Returning the service result information to the requesting end of the service request so that the requesting end can provide the target service; Wherein, the multimodal generation model at least includes a feedforward module, an attention module and an acceleration network, the acceleration network includes an acceleration module located before the target module and a restoration module located after the target module, and the target module is any one of the feedforward module and the attention module; wherein, the acceleration network adopts the method described in any one of claims 1-12 to perform model acceleration for the multimodal generation model.

15. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method of any one of claims 1 to 12, claim 13 and claim 14.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 1 to 12, claim 13 and claim 14.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps in the method of any one of claims 1 to 12, claim 13 and claim 14.

Images