Method for updating parameters in pre-training model and method for processing data of pre-training model

Abstract

The invention discloses a method for updating parameters in a pre-training model and processing data of the pre-training model, and relates to the field of large model technology and data processing. The method comprises the steps that feature data output by a backbone network of a pre-training model is acquired, and the backbone network is from an initial backbone network; an initial bypass network is called to convert the feature data, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; the parameters of the initial bypass network are updated based on the converted feature data, a target bypass network is obtained, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used for representing the influence condition of the tuning module on the feature data. The technical problems of high resource consumption and low calculation efficiency in the training process of the model are solved.

Description

Technical Field

[0001] The present invention relates to the field of large models and data processing, and in particular to a method for updating parameters in a pre-trained model and processing data of the pre-trained model. Background Art

[0002] At present, for efficient parameter migration of large-scale pre-trained models, different parts of the original training architecture of the large-scale pre-trained models are usually lightweight adjusted according to the different tasks performed by the pre-trained models. However, when training the pre-trained models, it is still necessary to perform redundant calculations on different tasks through the initial backbone network, which makes the model training process consume a lot of resources and leads to low efficiency in obtaining feedback results in generative dialogue products. As a result, there are technical problems such as high resource consumption and low computational efficiency in the model training process.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiment of the present invention provides a method for updating parameters in a pre-trained model and processing data of the pre-trained model, so as to at least solve the technical problems of high resource consumption and computational efficiency in the training process of the model.

[0005] According to one aspect of an embodiment of the present invention, a method for updating parameters in a pre-trained model is provided. The method may include: obtaining feature data output by a backbone network of the pre-trained model, wherein the backbone network comes from an initial backbone network; calling an initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; updating the parameters of the initial bypass network based on the converted feature data to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data.

[0006] According to another aspect of an embodiment of the present invention, a data processing method for a pre-trained model is also provided. The method may include: responding to an inquiry information received in a generative interactive interface, wherein the inquiry information at least includes: keywords of text generation information; calling a backbone network of a pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from an initial backbone network; calling a target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted text feature data, generating at least one reply result matching the inquiry instruction; and displaying the reply result in the generative interactive interface.

[0007] According to another aspect of an embodiment of the present invention, another data processing method for a pre-trained model is also provided. The method may include: obtaining feature data obtained by processing conditional data by a backbone network in a pre-trained model, wherein the backbone network comes from an initial backbone network, and the conditional data is used to determine the generation conditions of target data; calling a target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generating target data corresponding to the conditional data, wherein the type of the target data includes at least one of the following: text information, image information, video information, and voice information.

[0008] According to another aspect of an embodiment of the present invention, another data processing method for a pre-trained model is also provided. The method may include: inputting multimodal information in a dialogue interface, wherein the type of the multimodal information includes at least one of the following: text information containing character information, video frame information containing frame image information, and audio information; calling the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network; calling the target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generating reply information corresponding to the multimodal information, wherein the type of the reply information includes at least one of the following: text information, image information, video information, and voice information.

[0009] According to another aspect of an embodiment of the present invention, a data processing system for a pre-trained model is also provided. The system may include: a front-end client, which displays a dialogue interface and captures multimodal information input in the dialogue interface, wherein the type of multimodal information includes at least one of the following: query information containing character information, video frame information containing frame image information, and audio information; a back-end server, which is used to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and generate reply information corresponding to the multimodal information based on the converted feature data, wherein the backbone network comes from the initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of reply information includes at least one of the following: text information, image information, video information, and voice information.

[0010] According to another aspect of an embodiment of the present invention, another data processing method for a pre-trained model is also provided. The method may include: inputting a speech to be converted on a virtual reality VR device or an augmented reality AR device; extracting feature data from the speech to be converted using a backbone model in the pre-trained model, wherein the backbone network comes from an initial backbone network; calling a target bypass network in the pre-trained model to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, determining the image information corresponding to the speech to be converted; using the image information to activate the VR device or AR device, and displaying the image information in the VR device or AR device.

[0011] According to another aspect of an embodiment of the present invention, an electronic device is also provided, which may include a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions. When the above-mentioned computer-executable instructions are executed by the processor, the above-mentioned method of any one of the above-mentioned items is implemented.

[0012] According to another aspect of an embodiment of the present invention, a processor is further provided, the processor being used to run a program, wherein any one of the above methods is executed when the program is running.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above methods.

[0014] In an embodiment of the present invention, feature data output by a backbone network of a pre-trained model is obtained, wherein the backbone network comes from an initial backbone network; an initial bypass network is called to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; parameters of the initial bypass network are updated based on the converted feature data to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data. That is, in this embodiment, a tuning module extracted from the backbone network is used to construct the initial bypass network. At the same time, during the adjustment of the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) that is completely independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby saving memory and improving the training speed, thereby achieving the technical effect of reducing resource consumption during the training process of the model, and solving the technical problems of high resource consumption and computing efficiency during the training process of the model.

[0015] It is easy to be noted that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present invention, and do not constitute a limitation of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 is a schematic diagram of an application scenario of a method for updating parameters in a pre-training model according to an embodiment of the present invention;

[0018] Figure 2 is a flowchart of a method for updating parameters in a pre-training model according to an embodiment of the present invention;

[0019] Figure 3 is a flow chart of a data processing method of a pre-training model according to an embodiment of the present invention;

[0020] Figure 4 is a flowchart of another data processing method of a pre-training model according to an embodiment of the present invention;

[0021] Figure 5 is a flowchart of another data processing method of a pre-training model according to an embodiment of the present invention;

[0022] Figure 6is a schematic diagram of a data processing result of a pre-training model according to an embodiment of the present invention;

[0023] Figure 7 is a flowchart of another data processing method of a pre-training model according to an embodiment of the present invention;

[0024] FIG8( a ) is a schematic diagram of a data processing system for a pre-training model according to an embodiment of the present invention;

[0025] FIG8( b ) is a schematic diagram of a tuning module according to an embodiment of the present invention;

[0026] FIG8( c ) is a schematic diagram of another tuning module according to an embodiment of the present invention;

[0027] Fig. 9 is a schematic diagram of a pre-training model according to an embodiment of the present invention;

[0028] Fig.10 is a schematic diagram of another pre-training model according to an embodiment of the present invention;

[0029] Fig.11 is a schematic diagram of a device for updating parameters in a pre-training model according to an embodiment of the present invention;

[0030] Fig.12 is a schematic diagram of a data processing device for a pre-training model according to an embodiment of the present invention;

[0031] Fig.13 is a schematic diagram of another data processing device for a pre-training model according to an embodiment of the present invention;

[0032] Fig.14 is a schematic diagram of another data processing device for a pre-training model according to an embodiment of the present invention;

[0033] Fig.15 is a structural block diagram of a computer terminal according to an embodiment of the present invention;

[0034] Fig.16 The present invention is a block diagram of an electronic device for a method for updating parameters in a pre-training model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] The technical solution provided by the present invention is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0038] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In an embodiment of the present invention, data processing through a pre-trained model in scenarios such as machine intelligence technology, basic visual intelligence, and video artificial intelligence (AI) is used as an example for explanation.

[0039] First, some nouns or terms that appear in the process of describing the embodiments of the present invention are subject to the following explanations:

[0040] Pre-trained models can be produced after large-scale training and tuning on representative data sets. They can be used as initialization models for training other downstream tasks, and can be used to speed up training or achieve better results.

[0041] Foundation Model: A model trained in different fields using large amounts of data, powerful computing power, and a well-designed structure, which can be used to process downstream tasks.

[0042] Transfer Learning can improve the learning of new tasks by transferring knowledge from related tasks that have already been learned;

[0043] Parameter-efficient transfer learning (PETL) refers to a tuning training method that modifies a small number of parameters or adds a small number of additional parameters based on a pre-trained model.

[0044] Memory-efficient Transfer Learning (METL) is a method for fine-tuning training methods based on pre-trained models using relatively small amounts of memory.

[0045] The deep learning model (Transformer) can be a network architecture consisting of several layers of encoders and decoders;

[0046] Vision Transformer can be a network architecture, which can be a migration application of Transformer architecture in the visual field;

[0047] Submodule (TransformerBlock), which can be a submodule of Transformer, can be composed of multi-head attention and feedforward network;

[0048] Multi-head Attention (MHA) can be a submodule in a Transformer, which can be a module that calculates the related vectors of query, key, and value;

[0049] Feed-forward Network (FFN) can be a submodule in a Transformer and can be composed of multiple fully connected layers and activation functions;

[0050] Multi-Layer Perceptron (MLP) can be a network module, which can be composed of one or more hidden layers and activation functions;

[0051] Adapter, which can be a tuning training method, can act on a feedforward network and can contain a small module consisting of two fully connected layers and an activation function;

[0052] Prompt tuning method (Prompt) can be a tuning training method, which can refer to tuning training with the help of a learnable parameter spliced ​​with the input;

[0053] Prefix tuning can be a tuning method that uses two learnable parameters concatenated with the key and value in the multi-head attention layer to perform tuning training;

[0054] Discriminative tasks can refer to tasks that discriminate input data, such as discriminating Y based on X, for example, image classification tasks;

[0055] Generative tasks can be tasks that generate operations on input data, such as generating Y based on X, for example, image generation tasks;

[0056] Convolutional Neural Network (U-Net) can be a network architecture that can be composed of a downsampling encoder, an upsampling decoder, and skip connections;

[0057] The deep learning text-to-image generation model (StableDiffusion) can be a network structure for generation tasks, which can be composed of an autoencoder, a text encoder, and a U-Net structure.

[0058] Example 1

[0059] According to an embodiment of the present invention, a method for updating parameters in a pre-trained model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0060] Considering that the model parameters of the large model are huge and the computing resources of the mobile terminal are limited, the method for updating the parameters in the pre-trained model provided in the embodiment of the present invention can be applied to the following examples: Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of a method for updating parameters in a pre-training model according to an embodiment of the present invention. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface, for example, by inputting classification conditions in the graphical user interface, so as to realize the call of the large model, and then realize the method provided by the embodiment of the present invention.

[0061] In an embodiment of the present invention, a system composed of a client device and a server can perform the following steps: the client device obtains an image to be classified. The server executes step S101 to obtain feature data obtained by processing the image to be classified by the backbone network in the pre-trained model; step S102, calling the target bypass network to convert the feature data, wherein the target bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the backbone network; step S103, based on the converted feature data, determining the category to which the image to be classified belongs. The server 10 can transmit the category to which the image to be classified belongs to the client and display it in the graphical user interface of the client. It should be noted that when the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present invention can be performed in the client device.

[0062] Under the above operating environment, the present invention provides the following in the case of discriminant tasks: Figure 2 The method for updating parameters in the pre-trained model shown. Figure 2 FIG. 1 is a flow chart of a method for updating parameters in a pre-training model according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps.

[0063] Step S202, obtaining feature data output by a backbone network of the pre-trained model, wherein the backbone network comes from an initial backbone network.

[0064] In the technical solution provided in the above step S202 of the present invention, the backbone network can be obtained from the initial backbone network. Feature data output by the backbone network can be obtained. Among them, the pre-trained model can also be called a pre-trained network, which can be a model produced after large-scale training and tuning on a representative data set, which can be used as an initialization model when training other downstream tasks, and can be used to speed up the training speed or obtain better results. Feature data can also be called output features, which can be data obtained by the backbone network processing voice, image and other information, and can be represented by x0 and x1. This is only an example, and no specific restrictions are made on the representation of feature data. The backbone network can be used for tasks such as image classification, target detection and semantic segmentation, and can include feedforward network modules, multi-head attention modules, etc. It can be a visual task processing network (Vision Transformer, referred to as ViT), a convolutional neural network (Convolutional Neural Network, referred to as CNN), etc. This is only an example, and no specific restrictions are made on the content contained in the backbone network and the type of the backbone network. The feature data may be an abstract feature representation, which may be a vector, a pixel-level label, a bounding box, a category probability, etc. This is only an example and does not impose any specific restrictions on the type of feature data.

[0065] Optionally, the backbone network can output feature data during the forward propagation process.

[0066] For example, taking the image classification task as an example, the backbone network can be a convolutional neural network, and the input of the backbone network can be an image. After a series of convolutional layers, pooling layers, activation functions and other operations, a set of abstract feature representations is obtained.

[0067] Step S204, calling the initial bypass network to convert the characteristic data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

[0068] In the technical solution provided in the above step S204 of the present invention, at least one tuning module can be extracted from the backbone network, and the extracted at least one tuning module can be constructed to obtain an initial bypass network. The constructed initial bypass network can be called to convert the feature data. Among them, the tuning module can also be called a submodule, which can be extracted from the initial backbone network and can be used to improve the accuracy of the pre-trained model prediction. The initial bypass model can also be called a tuning framework (Residual Tuning, abbreviated as Res-Tuning), which can be represented by Bypass.

[0069] Optionally, a tuning module can be extracted from the initial backbone network, and a bypass network can be constructed using the tuning module to obtain an independent bypass network and backbone network. The feature data output by the backbone network can be used as the input of the initial bypass network, and the initial bypass network can be called to convert the feature data.

[0070] Step S206, updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data.

[0071] In the technical solution provided in the above step S206 of the present invention, the converted feature data can be output to the pre-trained model for outputting the corresponding results, and based on the output results, the parameters of the initial bypass network are adjusted to obtain the target bypass network. Among them, the target bypass network can be a new tuning network that is completely independent of the backbone network. The parameters of the initial bypass network can be used to characterize the influence of different tuning modules on the feature data, and can be weights, biases, regularization parameters, etc., which are only used as examples here, and the types of parameters are not specifically limited. The output results can be images, text, voice and other data recognized by the pre-trained model, which are only used as examples here, and the types of output results are not specifically limited. During the training process, the data flow of the initial bypass network is independent of the data flow of the backbone network, so the data flow of the initial bypass network cannot be transmitted to the backbone network.

[0072] Since the initial backbone network usually embeds the tuning module in the backbone network, the backbone network still needs to perform redundant calculations during each training process. For example, during the back propagation process, the gradient of the backbone network still needs to be further calculated, which wastes memory and inference overhead, and further leads to the technical problem that the training process of the pre-trained model consumes a long time. To solve the above problem, in this embodiment, the tuning module is extracted from the initial backbone network, and the initial bypass model is constructed using the tuning module to obtain an independent initial bypass network and backbone network. During the training of the initial bypass network, the data stream of the initial bypass network is prohibited from being transmitted to the backbone network, so as to independently train the initial bypass network and obtain the target bypass network, thereby achieving the technical effect of reducing resource consumption in the training process of the model and improving computing efficiency, and solving the technical problems of high resource consumption and computing efficiency in the training process of the model.

[0073] Optionally, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network can be prohibited from being transmitted to the backbone network, and the parameters of the initial bypass network can be adjusted based on the converted feature data through back propagation or stochastic gradient descent to obtain the target bypass network.

[0074] For example, the tuning modules can be extracted from the initial backbone network, and the extracted tuning modules can be combined and constructed to obtain the initial bypass network. The initial backbone network with the extracted tuning modules can be used as the backbone network to obtain the feature data output by the backbone network. The initial bypass network can be called to convert the feature data, and the converted feature data can be output to the pre-trained model to output the corresponding results. Based on the output results, the parameters of the initial bypass network can be adjusted to obtain the target bypass network.

[0075] In an embodiment of the present invention, a tuning module extracted from a backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (i.e., a new tuning module) that is completely independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby saving memory and improving the training speed, thereby achieving the technical effect of reducing resource consumption in the training process of the model and improving computing efficiency, and solving the technical problems of high resource consumption and computing efficiency in the training process of the model.

[0076] The above method of this embodiment is further introduced below.

[0077] As an optional implementation, step S204, calling the initial bypass network to convert the feature data, includes: calling the tuning module in the initial bypass network to adjust the feature data; and performing weighted summation on the adjusted feature data based on the weight of the initial bypass network.

[0078] In this embodiment, the tuning module in the initial bypass network can be called to adjust the feature data, and the adjusted feature data can be weighted summed based on the weight of the initial bypass network. The weight in the initial network bypass is continuously learned, and can be used to characterize the proportion of different tuning modules, and can also be called the λ coefficient, which can be in a state of continuous learning.

[0079] Optionally, during the training process of the initial bypass network, during forward propagation, the intermediate output of the backbone network (i.e., feature data) can be used as the input of the tuning module in the initial bypass network, the tuning module in the initial bypass network can be called to adjust the feature data, and the adjusted feature data can be weighted summed based on the parameters of the initial bypass network.

[0080] In this embodiment, a framework for synchronous combined training of a backbone network and an initial bypass network is used, and the initial bypass network is constructed by combining tuning modules extracted from the initial backbone network. During the forward propagation process of the initial bypass network in training, the feature data output by the backbone network is used, and during the reverse propagation process, the initial bypass network is disconnected from the backbone network, and only the initial bypass network is passed through, so that the gradient of the backbone network does not need to be further calculated, thereby achieving memory saving and improving training speed, thereby achieving the purpose of saving memory occupied during data processing, and obtaining a new target bypass network that is completely independent of the backbone network.

[0081] As an optional implementation, the tuning module in the initial bypass network is called to adjust the characteristic data, including: calling the first vertical tuning module corresponding to the first layer of the backbone network in the backbone network to adjust the characteristic data output by the zeroth layer of the backbone network, and calling the first horizontal tuning module associated with the first vertical tuning module to adjust the characteristic data output by the first layer of the backbone network.

[0082] In this embodiment, there is a corresponding relationship between the number of tuning modules in the initial bypass network and the number of layers of the backbone network. The first vertical tuning module corresponding to the first layer of the backbone network in the backbone network can be called to adjust the feature data output by the zeroth layer of the backbone network in the backbone network, and the first horizontal tuning module associated with the first vertical tuning module is called to adjust the feature data output by the first layer of the backbone network. Among them, the first layer of the backbone network can be a coding layer (Decoder), a convolutional layer (block), etc., which is only an example here, and there is no specific restriction on the category of the first layer of the backbone network. The zeroth layer of the backbone network can be an intermediate layer network (Middle), a convolutional layer, etc., which is only an example here, and there is no specific restriction on the category of the zeroth layer of the backbone network. The first vertical tuning module corresponds to the first layer of the backbone network, and can be a vertical tuning module, which can also be called a vertical Res-Tuner. The first horizontal tuning module is associated with the first vertical tuning module, and can be a horizontal tuning module, which can also be called a horizontal Res-Tuner.

[0083] Optionally, a tuning module (Res-Tuner) can be extracted from the initial bypass network, and a complete initial bypass network can be constructed by combining and designing different tuning modules. The first vertical tuning module corresponding to the first-layer backbone network in the initial bypass network can be called to adjust the feature data output by the zero-layer backbone network of the backbone network, and the first horizontal tuning module associated with the first vertical tuning module can be called to adjust the feature data output by the first-layer backbone network. The feature data output by the zero-layer backbone network can be represented by x0. The feature data output by the first-layer backbone network can be represented by x1.

[0084] As an optional implementation, based on the parameters of the initial bypass network, the adjusted feature data is weightedly summed, including: based on the weight of the initial bypass network, the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module are weighted summed to obtain the converted first feature data.

[0085] In this embodiment, the weight of the initial bypass network can be determined. The adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module are obtained. Based on the initial bypass network weight, the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module are weighted summed to obtain the converted first feature data.

[0086] As an optional implementation, updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: in response to the first layer backbone network being the last layer network in the backbone network, updating the parameters of the initial bypass network based on the first characteristic data to obtain the target bypass network.

[0087] In this embodiment, it is determined whether the first-layer backbone network is the last layer of the backbone network. If the first-layer backbone network is the last layer of the backbone network, the first feature data can be used as the output of the initial bypass network, and the first feature data can be used as the training data for back propagation to adjust the parameters of the initial bypass network to obtain the target bypass network.

[0088] Optionally, the overall structure of the pre-trained model can be divided into two parts, one part is the backbone model, the parameters of which are frozen during the training process; the other part is the trainable initial bypass network (also referred to as the bypass structure), which can be composed of several tuning modules. Each layer of the backbone network has a horizontal tuning module and a vertical tuning module, which are respectively used to receive the intermediate feature data from the layer in the backbone network and the output data of the previous layer in the bypass network, and perform weighted summation and output to the next layer, until the last layer of features is output to the pre-trained model (for example, the discriminative task network) for the output of the corresponding results, and the output results are obtained. Based on the output results, the parameters of the initial bypass network can be updated to obtain the target bypass network.

[0089] For example, training data (e.g., labeled images, speech, and other data) can be obtained, and the backbone network can be obtained to process the training data and output feature data. The first vertical tuning module corresponding to the first layer of the backbone network in the initial bypass network can be called to adjust the feature data output by the zeroth layer of the backbone network, and the first horizontal tuning module corresponding to the first vertical tuning module can be called to adjust the feature data output by the first layer of the backbone network. Based on the weight of the initial bypass network, the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module are weighted summed to obtain the converted first feature data, and it is determined whether the first layer of the backbone network is the last layer of the backbone network. If the first layer of the backbone network is the last layer of the backbone network, the first feature data can be used as the output of the initial bypass network. The first feature data can be input into the output layer (e.g., Head) in the pre-trained model, and the output layer converts the first feature data to obtain the final training result. Back propagation calculation can be performed based on the training result to update the parameters of the initial bypass network to obtain the target bypass network.

[0090] As an optional implementation, updating the parameters of the initial bypass network based on the converted feature data to obtain the target bypass network includes: in response to the last layer of the network in the non-backbone network of the first layer backbone network, calling the second vertical tuning module corresponding to the second layer backbone network in the backbone network to adjust the first feature data, and calling the second horizontal tuning module associated with the second vertical tuning module to adjust the feature data of the second layer backbone network in the backbone network; based on the weight of the initial bypass network, performing weighted summation between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the converted second feature data; in response to the last layer of the network in the non-backbone network of the second layer backbone network, executing the following steps: The invention discloses a method for transmitting a bypass network to a plurality of network components, comprising: determining a second-layer backbone network as a first-layer backbone network, and determining a third-layer backbone network in the backbone network as a second-layer backbone network; calling a second vertical tuning module to adjust the first feature data, and calling a second horizontal tuning module to adjust the feature data of the second-layer backbone network; based on the weight of the initial bypass network, performing weighted summation between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the converted second feature data, until the second-layer backbone network is the last layer of the backbone network; updating the parameters of the initial bypass network based on the converted feature data to obtain the target bypass network, including: updating the parameters of the initial bypass network based on the second feature data to obtain the target bypass network.

[0091] In this embodiment, a weighted sum is performed between the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module to obtain the converted first feature data, and it is determined whether the first-layer backbone network is the last layer of the backbone network. If the first-layer backbone network is not the last layer of the backbone network, the first feature data can be used as input data of the second vertical tuning module corresponding to the second-layer backbone network, and the second vertical tuning module corresponding to the second-layer backbone network in the backbone network can be called to adjust the first feature data, and the second horizontal tuning module associated with the second vertical tuning module can be called to adjust the feature data of the second-layer backbone network in the backbone network. Based on the weight of the initial bypass network, a weighted sum can be performed between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the converted second feature data.

[0092] Optionally, this embodiment determines whether the second-layer backbone network is the last layer of the backbone network. If the second-layer backbone network is the last layer of the backbone network, the second feature data can be output to the output layer of the pre-trained model. The output layer converts the second feature data to obtain the final output result. Back propagation calculation can be performed based on the output result to update the parameters of the initial bypass network to obtain the target bypass network. If the second-layer backbone network is not the last layer of the backbone network, the second-layer backbone network can be determined as the first-layer backbone network, and the third-layer backbone network in the backbone network can be determined as the second-layer backbone network; the second vertical tuning module is called to adjust the first feature data, and the second horizontal tuning module is called to adjust the feature data of the second-layer backbone network; based on the weight of the initial bypass network, the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module are weighted summed to obtain the converted second feature data, until the second-layer backbone network is the last layer of the backbone network, the obtained second feature data is output to the output layer of the pre-trained model, and the output result of the output layer is back-propagated to update the parameters of the initial bypass network to obtain the target bypass network.

[0093] Optionally, the second characteristic data output by the bypass network may be determined based on the following formula:

[0094]

[0095]

[0096] in, It can be used to characterize the adjusted feature data corresponding to the first-level horizontal tuning module. It can be used to characterize the output of the bypass network of layer l, Res-Tuner(x l ) can be used to characterize the output of the level tuning module at layer l, It can be used to characterize the output of the vertical tuning module at the l-1th layer, x0 can be used to characterize the feature data output by the backbone network at the zeroth layer, and x l It can be used to characterize the output features of the lth layer, and λ can be used to characterize the weight of the initial bypass network.

[0097] Optionally, the weight of the initial bypass network may be a learnable weight coefficient, which may be continuously changed during the training of the initial bypass network.

[0098] Optionally, in each network layer corresponding to the backbone network, the initial bypass network uses a horizontal tuning module and a vertical tuning module, respectively, to process the feature data output from the lth layer of the backbone network and the first feature data after conversion from the l-1th layer of the bypass network, and the feature data of the two can be weighted and summed using weights.

[0099] Optionally, the initial input into the entire module is called layer 0.

[0100] As an optional implementation, step S206, updating the parameters of the initial bypass network based on the converted feature data to obtain the target bypass network, includes: determining the processing result obtained by the output layer in the pre-trained model processing the converted feature data; determining the difference between the processing result and the actual processing result corresponding to the processing result; adjusting the parameters of the initial bypass network based on the difference value to obtain the target bypass network.

[0101] In this embodiment, feature data after conversion of the initial bypass network is obtained, and the feature data can be processed through the output layer in the pre-trained network to obtain a processing result. The difference value between the processing result and the real processing result corresponding to the processing result can be determined. Based on the difference value between the two, the parameters of the initial bypass network can be adjusted to obtain the target bypass network. Among them, the processing result can be an image, text, sequence data, classification situation, etc., and the category of the processing result can be flexibly changed according to the use scenario of the pre-trained model. This is only an example, and no specific limitation is made to the category of the processing result. The difference value can be used to characterize the difference between the processing result and the real processing result corresponding to the processing result, and can be used to determine the loss function.

[0102] Optionally, this embodiment obtains feature data after conversion of the initial bypass network, and can process the feature data through the output layer in the pre-trained network to obtain a processing result. The loss function between the processing result and the actual processing result corresponding to the processing result can be calculated, and the gradient of the parameters of the initial bypass network is calculated based on the loss function, thereby determining the adjustment direction and size of the parameters to obtain an updated target bypass network.

[0103] For example, in order to make the model better fit the training data, back propagation can be performed based on the processing results of the output layer, and the change of the loss function can be determined according to the processing results and the actual processing results. The parameters of the initial bypass network can be adjusted based on the change of the loss function, so that the processing results of the pre-trained model are closer to the actual processing results.

[0104] Due to the large number of parameters in the large model, it is very time-consuming to directly calculate the gradient through numerical methods. In this embodiment, the gradient can be efficiently calculated through back propagation, which speeds up the training of the initial bypass network. In addition, back propagation also enables the initial bypass network to learn the characteristics and laws of the data, and continuously optimizes the performance of the pre-trained model by updating the parameters of the initial bypass network, thereby improving the accuracy and generalization ability of the pre-trained model.

[0105] As an optional implementation, the backbone network and at least one tuning module are connected based on residuals.

[0106] In this embodiment, the initial bypass network and the backbone network can be connected by residual connection, and the initial bypass network can include at least one horizontal tuning module or a vertical tuning module, so the backbone network and at least one tuning module can be connected based on residual. It should be noted that there is no specific restriction on the number of tuning modules in the initial bypass network.

[0107] Optionally, the tuning modules in the initial bypass network can be connected based on residual connections and act in parallel in the backbone network.

[0108] Since the initial backbone network constructed by embedding the tuning module in the backbone network needs to process the training data during the back propagation training of the tuning module, there is a problem of occupying memory during the training process. In this embodiment, the tuning module is extracted from the initial backbone network, and the tuning modules are connected through residual connections to construct an initial bypass network, and the initial bypass network can be connected to the backbone network in parallel. During the back propagation process, it is prohibited to transmit the data stream from the initial bypass network to the backbone network, thereby achieving the effect of reducing memory usage.

[0109] It should be noted that the above-mentioned residual connection can be changed according to actual conditions. For example, it can also be a full connection, a skip connection, etc. Different units can also be combined or replaced inside a single tuning module. No specific restrictions are made here on the connection method between the tuning modules and the components of the tuning modules.

[0110] As an optional implementation, during the process of transmitting the characteristic data to the initial bypass network, the state of the parameters of the backbone network is a locked state.

[0111] In this embodiment, during the model training process, the state of the parameters of the backbone network is a locked state, that is, the parameters of the backbone network are not updated. The locked state can also be called a frozen state, which can be used to characterize that the parameters of the backbone network do not change.

[0112] In this embodiment, the backbone network keeps parameters frozen, and at the same time, data no longer flows to the backbone network during the back propagation process, thereby achieving the purpose of saving memory.

[0113] In an embodiment of the present invention, a tuning module extracted from a backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) that is completely independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby achieving memory savings and improved training speed, thereby achieving the technical effect of reducing resource consumption during the training process of the model, and solving the technical problems of high resource consumption and computational efficiency during the training process of the model.

[0114] The embodiment of the present invention is directed to an Artificial Intelligence Generated Content (AIGC) system and also provides a data processing method for a pre-trained model. Figure 3 is a flow chart of a data processing method of a pre-training model according to an embodiment of the present invention. Figure 3 As shown, the method may include the following steps.

[0115] Step S302, responding to the query information received in the generative interactive interface, wherein the query information at least includes: keywords of the text generation information.

[0116] In the technical solution provided in the above step S302 of the present invention, when the generative interaction interface receives the inquiry information, the inquiry information received in the generative interaction interface can be responded to. Among them, the generative interaction interface can be an interface that generates a dialogue using a natural speech generation model, and can be an operation interface of a mobile terminal, a client, etc. This is only an example, and there is no specific restriction on the type of generative interaction interface. The inquiry information may at least include: keywords of the text generation information. The inquiry information may be an instruction issued by the client, and may be an instruction to ask questions in a dialogue or interaction scenario to obtain guidance behavior, etc.

[0117] Optionally, the user may enter a question or request in the generative interactive interface, "Hello, I would like to know how to unsubscribe from my service?", and the query information may include the keyword "how" of the text generation information. The query information received in the generated dialogue interface may be responded to.

[0118] Step S304, calling the backbone network of the pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from the initial backbone network.

[0119] In the technical solution provided in the above step S304 of the present invention, in response to the query information received in the generative interactive interface, the backbone network of the pre-trained model can be called to at least analyze the text generation information and output text feature data. Among them, the backbone network can come from the initial backbone network. The text generation information can be used to determine the question raised by the query information. The text feature data can be used to characterize the text content, for example, it can be a vector, texture feature, etc., which is only for example description and does not specifically limit the type of feature data.

[0120] Step S306, calling a target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

[0121] In the technical solution provided in the above step S306 of the present invention, the target bypass network can be called to convert the text feature data.

[0122] Step S308: generating at least one response result matching the query information based on the converted text feature data.

[0123] In the technical solution provided in the above step S308 of the present invention, the pre-trained model can process the converted text feature data to generate a response result that matches the query information.

[0124] Step S310, displaying the reply result in the generative interactive interface.

[0125] In the technical solution provided in the above step S310 of the present invention, the reply result can be displayed in the generative interactive interface. The reply result can be an image, text, voice, etc., which is only an example and does not specifically limit the content of the reply result.

[0126] For example, a generative interactive interface receives a query message such as “What is the recipe for pasta?” In response to the query message received by the generative interactive interface, the backbone network in the pre-trained model can be called to at least analyze the text generation information “recipe for pasta” in the query message, output text feature data, and call the target bypass network to convert the text feature data. Based on the converted text feature data, at least one reply result matching the query message is generated, and the reply result can be displayed in the generative interactive interface.

[0127] As an optional implementation, the method also includes: in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the impact of the tuning module on the text feature data.

[0128] In this embodiment, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters in the initial bypass network can be used to characterize the influence of the tuning module on the text feature data.

[0129] Optionally, the backbone network keeps parameters frozen, and the initial bypass network uses a tuning module extracted from the backbone network, while truncating the data flow from the tuning module to the backbone network, thereby forming an initial bypass network that is completely independent of the backbone network.

[0130] In this embodiment, a query information received in a generative interactive interface is responded to, wherein the query information includes at least: keywords of text generation information; a backbone network of a pre-trained model is called to at least analyze the text generation information and output text feature data, wherein the backbone network comes from an initial backbone network; a target bypass network is called to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of an initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted text feature data, at least one reply result matching the query information is generated; and the reply result is displayed in the generative interactive interface, thereby achieving the technical effect of improving the efficiency of obtaining feedback results in a generative dialogue product and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

[0131] As an optional embodiment, in the application scenario of discriminative tasks, this embodiment can extract the tuning module from the initial backbone network to obtain the tuning module and the backbone network. The extracted tuning modules can be combined to obtain the initial bypass network, and the parameters in the initial bypass network can be updated to obtain the target bypass network. In the scenario of discriminative tasks, the backbone network in the pre-trained model can be obtained to process the image to be classified and obtain feature data. The trained target bypass network is called to convert the feature data. The feature data converted by the target bypass model can be input into the output layer, and the output layer processes the feature data to obtain the category to which the image to be classified belongs.

[0132] Optionally, in a discriminative task, the overall structure of the pre-trained model (e.g., a discriminative task network) can be divided into two parts. One part is the backbone network, which is frozen during the training process. The other part is a trainable initial bypass network (also referred to as a bypass structure), which can be composed of several tuning modules, wherein each network layer of the corresponding backbone network has a horizontal tuning module and a vertical tuning module, which are respectively used to receive feature data output from the layer in the backbone network and the output of the previous layer in the initial bypass network, and the two are weighted summed and output to the next layer.

[0133] Optionally, until the converted feature data of the last layer is output to the output layer in the pre-trained model to output the corresponding result, a back propagation process can be performed based on the output of the corresponding result, and during the back propagation process, the back propagation path from the bypass network to the backbone network will be disconnected to obtain the trained target bypass grid. When the image to be classified is obtained, the backbone model can be obtained to process the image to be classified and obtain the feature data. The target bypass network can be called to convert the feature data, and the converted feature data can be output to the output layer in the pre-trained model to determine the category to which the image to be classified belongs.

[0134] For example, given an image to be classified, the feature data of the image to be classified can be determined through the backbone network in the pre-trained model, the target bypass network can be called to convert the feature data, and based on the converted feature data, the category of the image to be classified is determined to be an image containing kittens.

[0135] As an optional implementation, step S404, calling the target bypass network to convert the text feature data, includes: calling the first vertical tuning module corresponding to the first decoding layer in the backbone network to convert the text feature data output by the middle layer of the backbone network, and calling the first horizontal tuning module associated with the first vertical tuning module to convert the text feature data output by the first decoding layer.

[0136] In this embodiment, the first vertical tuning module corresponding to the first decoding layer in the backbone network can be called to convert the text feature data output by the middle layer of the backbone network, and the first horizontal tuning module associated with the first vertical tuning module can be called to convert the text feature data output by the first decoding layer.

[0137] Optionally, in a generative task (e.g., image generation), the overall structure of the pre-trained model may include a backbone network whose parameters are frozen during training and a target bypass network whose parameters are updated, wherein the backbone network may be a deep learning model (U-Net) structure based on stable diffusion (StableDiffusion). The target bypass network may be a trainable bypass structure, which may be composed of several tuning modules.

[0138] For example, the backbone network may include three encoding layers, one middle layer and four decoding layers. Each decoding layer corresponds to a horizontal tuning module and a vertical tuning module, and the horizontal tuning module and the vertical tuning module can be used to receive text feature data output from the middle layer or the decoding layer and the text feature data converted from the previous layer in the target bypass network. The first vertical tuning module corresponding to the first decoding layer can be used to receive the text feature data of the middle layer of the backbone network and convert the text feature data of the middle layer. The first horizontal tuning module associated with the first vertical tuning module can obtain the text feature data output by the first decoding layer and convert the text feature data output by the first decoding layer.

[0139] As an optional implementation, based on the converted text feature data, at least one response result matching the query instruction is generated, including: based on the weight of the target bypass network, performing weighted summation between the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain the converted text feature data; based on the converted text feature data, determining the response result.

[0140] In this embodiment, based on the weight of the target bypass network, a weighted sum can be performed on the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain converted text feature data, and a response result can be determined based on the converted text feature data.

[0141] Optionally, each corresponding decoding layer of the backbone network corresponds to a horizontal tuning module and a vertical tuning module. The horizontal tuning module and the vertical tuning module can respectively receive the text feature data output from the middle layer or the decoding layer in the backbone network and the text feature data output from the previous layer in the target bypass network, and can perform weighted summation on the two and output them to the vertical tuning module corresponding to the next layer of the backbone network, until the weighted summation result of the horizontal tuning module and the vertical tuning module corresponding to the last decoding layer of the backbone network is output to the generative task network for outputting the corresponding result, so as to obtain the response result.

[0142] For example, the backbone network may include three encoding layers, one intermediate layer and three decoding layers. Each decoding layer corresponds to a horizontal tuning module and a vertical tuning module, which can be used to receive text feature data output from the intermediate layer or the decoding layer and text feature data converted from the previous layer in the target bypass network. The first vertical tuning module corresponding to the first decoding layer can be used to receive the text feature data of the intermediate layer of the backbone network and convert the text feature data of the intermediate layer. The first horizontal tuning module associated with the first vertical tuning module can obtain the text feature data output by the first decoding layer and convert the text feature data output by the first decoding layer. Based on the weight of the target bypass network, a weighted sum can be performed between the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module. The text feature data obtained by weighted summing the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module can be input into the second vertical tuning module corresponding to the second decoding layer to obtain the text feature data converted by the second vertical tuning module, and the text feature data output by the second decoding layer is output to the second horizontal tuning module to obtain the corresponding text feature data adjusted by the second horizontal tuning module, and the text feature data adjusted by the second horizontal tuning module and the text feature data adjusted by the second vertical tuning module can be weighted summed. The text feature data obtained by weighted summing the text feature data adjusted by the second horizontal tuning module and the text feature data adjusted by the second vertical tuning module are continuously transmitted to the third vertical tuning module corresponding to the third decoding layer, and the text feature data output by the third decoding layer is transmitted to the third horizontal tuning module corresponding to the third decoding layer, and the text feature data adjusted by the third horizontal tuning module and the text feature data adjusted by the third vertical tuning module can be weighted summed to obtain the text feature data finally converted by the target bypass network. The text feature data can be processed based on the output layer in the pre-trained model to obtain the response result.

[0143] As an optional implementation, step S406, based on the converted text feature data, determines the reply result, including: using the converted text feature data as the output data of the last decoding layer in the pre-trained model; converting the output data into the reply result.

[0144] In this embodiment, the target bypass model can be used as the last encoding layer in the pre-trained model, and the converted text feature data can be used as the output data of the last decoding layer in the pre-trained model. Based on the output data, the output layer can convert the output data to obtain a response result.

[0145] Optionally, the target bypass network in this embodiment can also be used as a branch in a pre-trained model (e.g., a controllable generative task), that is, a coding module for conditional input. The controllable generative task can be a task generated based on controllable conditional data.

[0146] In this embodiment, the controllable coding part can also use the target bypass network as a branch to achieve the purpose of reducing memory consumption.

[0147] In this embodiment, feature data obtained by processing an image to be classified by a backbone network in a pre-trained model is obtained, wherein the backbone network comes from an initial backbone network; a target bypass network is called to convert the feature data, wherein the target bypass network is obtained by updating parameters of an initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, the category to which the image to be classified belongs is determined, thereby achieving a technical effect of reducing resource consumption in the training process of the model, and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

[0148] The embodiment of the present invention provides another data processing method for a pre-trained model for generating task scenarios, such as generating images, videos or texts according to conditional data. Figure 4 FIG. 1 is a flow chart of another method for processing data of a pre-trained model according to an embodiment of the present invention. Figure 4 As shown, the method may include the following steps:

[0149] Step S402, obtaining feature data obtained by processing the conditional data by the backbone network in the pre-trained model, wherein the backbone network comes from the initial backbone network, and the conditional data is used to determine the generation condition of the target data.

[0150] In the technical solution provided in the above step S402 of the present invention, conditional data can be obtained. The backbone network processes the conditional data to obtain feature data. Among them, the conditional data can be used to determine the generation conditions of the target data, for example, it can be "drawing a blue dwarf cat", it can be a line drawing, a depth map, a posture and other contents, and can include text conditions, image conditions, etc. This is only for example, and no specific restrictions are made on the content of the conditional data. The backbone network can be a conditional two-dimensional deep learning model (Conditional 2D U-Net), which can include an encoding layer (Encoder), a decoding layer (Docoder) and an intermediate layer (Middle).

[0151] For example, the conditional data "write a set of codes for filtering data" can be obtained. Keywords in the conditional data can be extracted and converted, and feature data corresponding to the keywords can be obtained. It should be noted that the content of the above conditional data and the determination of the feature data are only for illustration and are not specifically limited here.

[0152] Step S404, calling a target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

[0153] In the technical solution provided in the above step S404 of the present invention, a tuning module can be extracted from the backbone network, an initial bypass network can be constructed based on at least one tuning module, and the parameters of the initial bypass network can be updated based on the training data to obtain a target bypass network. The trained target bypass network can be called to convert the feature data. Among them, the training data can be data with labels, and can be pre-acquired data used to train the initial bypass network, for example, image data, text data, voice data, etc., which is only for example and does not specifically limit the category of the training data. The parameters of the initial bypass network can be weights, bias data, etc., which are only for example and do not specifically limit the type of parameters.

[0154] Optionally, this embodiment extracts a tuning module from the initial backbone network to obtain a tuning module and a backbone network. The extracted tuning modules can be combined to obtain an initial bypass network, and the parameters in the initial bypass network can be updated to obtain a target bypass network. In the scenario of generating tasks, the conditional data of the backbone network in the pre-trained model can be obtained to process the feature data. The trained target bypass network is called to convert the feature data.

[0155] Step S406: generating target data corresponding to the conditional data based on the converted feature data, wherein the type of the target data includes at least one of the following: text information, image information, video information and voice information.

[0156] In the technical solution provided in the above step S406 of the present invention, the feature data converted by the target bypass model can be input into the output layer, and the output layer processes the feature data to generate target data corresponding to the conditional data. The target data can be an animation, a story, a landscape, a face image, a dialogue, etc. generated based on the conditional data, and the type of the target data can include at least one of the following: text information, image information, video information, and voice information. This is only for example, and the category of the target data is not specifically limited.

[0157] Optionally, the pre-trained model can be a conditional pre-trained transformer model (Chat Conditional Pretrained Transformer, referred to as ChatCPT), which can be used to generate dialogue replies and conduct dialogue interactions. When the conditional data is the dialogue context, the inquiry information can be determined based on the dialogue context. The backbone network in the pre-trained model can be called to process the conditional data, and the feature data can be obtained. The target bypass network is called to transform the feature data; based on the converted feature data, a dialogue reply corresponding to the conditional data (i.e., target data) can be generated. Alternatively, when the conditional data is a question raised by the user, the pre-trained model can be used to generate an answer corresponding to the user's question information. Alternatively, when the conditional data is a long text message and the user wants to simplify the text information, a concise and general summary content can be generated through the pre-trained model. Alternatively, when the conditional data is information and guidance provided by the user, the pre-trained model can assist in writing, and the conditional data can be processed to obtain articles, storylines, dialogue scripts, and other content corresponding to the field data.

[0158] It should be noted that the above is only an example of the use scenario of the pre-training model, and no specific limitation is made here. The method of processing conditional data through a pre-training model to obtain target data should be within the scope of protection of the present invention.

[0159] In an embodiment of the present invention, a backbone network in a pre-trained model is obtained to process conditional data to obtain feature data, wherein the backbone network comes from an initial backbone network, and the conditional data is used to determine the generation condition of target data; a target bypass network is called to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, target data corresponding to the conditional data is generated, wherein the type of the target data includes at least one of the following: text information, image information, video information and voice information, thereby achieving the technical effect of reducing resource consumption in the training process of the model, and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

[0160] According to an embodiment of the present invention, there is also provided a data processing method for a pre-trained model in a virtual reality scenario such as a virtual reality VR device and an augmented reality AR device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0161] Figure 5 is a flowchart of another data processing method of a pre-training model according to an embodiment of the present invention. Figure 5 As shown, the method may include the following steps.

[0162] Step S502: input the speech to be converted on a virtual reality VR device or an augmented reality AR device.

[0163] Step S504, using the backbone model in the pre-trained model to extract feature data from the speech to be converted, wherein the backbone network comes from the initial backbone network.

[0164] Step S506, calling the target bypass network in the pre-trained model to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

[0165] Step S508: determining the image information corresponding to the speech to be converted based on the converted feature data.

[0166] Step S510: activating the VR device or AR device using the image information, and displaying the image information in the VR device or AR device.

[0167] Optionally, in this embodiment, the data processing method of the pre-trained model can be applied to a hardware environment composed of a server and a virtual reality device. The VR device or AR device is controlled to perform a human-computer interaction operation corresponding to the positioning information. The server can be a server corresponding to the media file operator. The network includes but is not limited to: a wide area network, a metropolitan area network or a local area network. The virtual reality device is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc.

[0168] Optionally, the virtual reality device may include: a memory, a processor, and a transmission device. The memory is used to store an application, which can be used to execute: inputting a speech to be converted on a virtual reality VR device or an augmented reality AR device; extracting feature data from the speech to be converted using a backbone model in a pre-trained model, wherein the backbone network comes from an initial backbone network; calling a target bypass network in the pre-trained model to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; determining image information corresponding to the speech to be converted based on the converted feature data; using the image information to activate the VR device or AR device, and displaying the image information in the VR device or AR device.

[0169] It should be noted that the processing method of the pre-trained model applied in the VR device or AR device in this embodiment may include: Figure 5 The method of the illustrated embodiment is used to control a VR device or an AR device to perform a human-computer interaction operation corresponding to the positioning information.

[0170] Optionally, the processor of this embodiment can call the application stored in the memory to execute the above steps through a transmission device. The transmission device can receive media files sent by the server through the network, and can also be used for data transmission between the processor and the memory.

[0171] Optionally, in a virtual reality device, a head-mounted display with eye tracking is provided, wherein the screen in the HMD is used to display the displayed video images, the eye tracking module in the HMD is used to obtain the real-time movement path of the user's eyes, the tracking system is used to track the user's position information and motion information in the real three-dimensional space, and the computing processing unit is used to obtain the user's real-time position and motion information from the tracking system, and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view direction in the virtual three-dimensional space, etc.

[0172] In an embodiment of the present invention, a virtual reality device may be connected to a terminal, and the terminal and the server are connected through a network. The virtual reality device is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc. The terminal is not limited to a PC, a mobile phone, a tablet computer, etc. The server may be a server corresponding to a media file operator, and the network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.

[0173] Figure 6 is a schematic diagram of a data processing result of a pre-training model according to an embodiment of the present invention, such as Figure 6 As shown, the speech to be converted "Draw a deer" is displayed on the presentation screen of the VR device or AR device, the backbone model in the pre-trained model can be used to extract feature data from the speech to be converted, and the target bypass network in the pre-trained model is called to convert the feature data, and based on the converted feature data, the image information corresponding to the speech to be converted can be determined; the image information can be used to activate the VR device or AR device, and the image information "deer" can be displayed in the VR device or AR device.

[0174] Through the above steps, the speech to be converted is input on the virtual reality VR device or the augmented reality AR device; the backbone model in the pre-trained model is used to extract feature data from the speech to be converted, wherein the backbone network comes from the initial backbone network; the target bypass network in the pre-trained model is called to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, the image information corresponding to the speech to be converted is determined; the image information is used to activate the VR device or AR device, and the image information is displayed in the VR device or AR device, thereby solving the technical problems of high resource consumption and computing efficiency in the training process of the model, and achieving the technical effect of reducing resource consumption in the training process of the model.

[0175] According to an embodiment of the present invention, there is also provided a data processing method for a pre-trained model that can be applied to an artificial intelligence generated content (AIGC) system. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0176] Figure 7 is a flowchart of another data processing method of a pre-training model according to an embodiment of the present invention. Figure 7As shown, the method may include the following steps.

[0177] Step S702: input multimodal information in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: inquiry information including character information, video frame information including frame image information, and audio information.

[0178] In the technical solution provided in step S702 of the present invention, multimodal information input in the dialogue interface can be obtained, wherein the type of multimodal information can include at least one of the following: query information containing character information, video frame information containing frame image information, and audio information. It should be noted that this is only an example and does not specifically limit the type of multimodal information. The dialogue interface can be the display interface of the terminal where the artificial intelligence content generation system is located. This is only an example. The dialogue interface that can be used for the dialogue system should be within the scope of protection of this application and is not specifically limited here.

[0179] Step S704, calling the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network.

[0180] In the technical solution provided in the above step S704 of the present invention, the AIGC system can obtain multimodal information input in the dialogue interface, and call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data.

[0181] Step S706, calling the target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

[0182] Step S708, generating reply information corresponding to the multimodal information based on the converted feature data, wherein the type of the reply information includes at least one of the following: text information, image information, video information and voice information.

[0183] In the technical solution provided in the above step S708 of the present invention, the pre-trained model processes the multimodal information to obtain converted feature data. Based on the converted feature data, reply information corresponding to the multimodal information can be generated, wherein the type of the reply information includes at least one of the following: text information, image information, video information and voice information. It should be noted that there is no specific restriction on the type of the reply information here.

[0184] For example, to obtain the multimodal information input by the user in the dialogue interface: "Give examples of usage scenarios of neural networks", the trained pre-trained model can be called to process the multimodal information to obtain response information corresponding to the multimodal information: "The usage scenarios of neural networks can be image usage scenarios, voice usage scenarios, and code direction usage scenarios."

[0185] As an optional implementation, step S706, calling the target bypass network to convert the characteristic data, includes: calling the tuning module in the target bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the target bypass network.

[0186] In this embodiment, the multimodal information input by the user in the dialogue interface can be obtained, and the backbone network in the pre-trained model can be called to analyze and process the multimodal information to obtain feature data. The feature data can be input into the trained target bypass network, and the tuning module in the target bypass network adjusts the feature data, and the adjusted feature data is weighted and summed based on the target bypass network to obtain the final converted feature data.

[0187] As an optional implementation, step S708, based on the converted feature data, generates reply information corresponding to the multimodal information, including: processing the converted feature data based on the output layer in the pre-trained model to obtain reply information corresponding to the multimodal information.

[0188] In this embodiment, the converted feature data may be processed based on the output layer in the pre-trained model to convert the feature data into reply information corresponding to the multimodal information. The reply information may be displayed in the dialogue interface.

[0189] In this embodiment, multimodal information is input in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: inquiry information containing character information, video frame information containing frame image information, and audio information; the backbone network in the pre-trained model is called to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network; the target bypass network is called to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, reply information corresponding to the multimodal information is generated, wherein the type of the reply information includes at least one of the following: text information, image information, video information, and voice information, thereby solving the technical problems of high resource consumption and computational efficiency in the training process of the model, and achieving the technical effect of reducing resource consumption in the training process of the model.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0191] For the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the order of the actions described, because according to the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0193] Example 2

[0194] According to an embodiment of the present application, an embodiment of a data processing system for a pre-trained model is also provided. Figure 8(a) is a schematic diagram of a data processing system for a pre-trained model according to an embodiment of the present invention. As shown in Figure 8(a), the data processing system 800 for a pre-trained model may include: a front-end client 802 and a back-end server 804.

[0195] The front-end client 802 is used to display the dialogue interface and capture multimodal information input in the dialogue interface, wherein the type of multimodal information includes at least one of the following: query information containing character information, video frame information containing frame image information, and audio information.

[0196] Optionally, multimodal information may be input in a dialogue interface of the front-end client 802 .

[0197] The back-end server 804 is used to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and generate reply information corresponding to the multimodal information based on the converted feature data, wherein the backbone network comes from the initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of the reply information includes at least one of the following: text information, image information, video information and voice information.

[0198] Optionally, the front-end client 802 may transmit the acquired multimodal information to the back-end server 804. The back-end server 804 may call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and generate reply information corresponding to the multimodal information based on the converted feature data.

[0199] Optionally, the backend server 804 may transmit the reply information to the dialogue interface of the frontend client 802 for display.

[0200] In this embodiment, a dialogue interface is displayed through a front-end client, and multimodal information input in the dialogue interface is captured, wherein the type of multimodal information includes at least one of the following: query information including character information, video frame information including frame image information, and audio information; through a back-end server, a backbone network in a pre-trained model is called to at least analyze and process the multimodal information to obtain feature data, and a target bypass network is called to convert the feature data, and based on the converted feature data, reply information corresponding to the multimodal information is generated, wherein the backbone network comes from an initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of reply information includes at least one of the following: text information, image information, video information, and voice information, thereby achieving a technical effect of reducing resource consumption in the training process of the model, and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

[0201] Example 3

[0202] At present, parameter-efficient transfer learning methods based on large-scale pre-trained basic models have achieved great success in various downstream applications. Existing tuning methods usually introduce lightweight and trainable structures to perform fine-tuning for specific tasks while freezing the basic model. However, most existing parameter-efficient transfer learning methods change the intermediate state of the pre-trained model, so more memory and time are required for training, and redundant calculations of different tasks need to be performed through the backbone during multi-task reasoning, resulting in the problem of high consumption and low computational efficiency of the training process of the pre-trained model.

[0203] As an optional embodiment, FIG8(b) is a schematic diagram of a tuning module according to an embodiment of the present invention. As shown in FIG8(b), the original tuning method will adjust the different parts of the original training architecture in a lightweight manner (such as Prompt, Prefix, Adapter, etc.) according to the specific tasks. However, this method deeply embeds the parameter-efficient tuning module into the original backbone network, where the original backbone network (initial backbone network) can include a feedforward network and multi-head attention. During training, different tasks need to be calculated through the initial backbone network, resulting in the problem of wasting memory and overhead during inference.

[0204] To solve the above problems, this embodiment proposes an efficient tuning method for parameters and memory based on a pre-trained model. This method constructs a bypass network (also called a tuning framework) with efficient parameters and memory, extracts the tuning module from the backbone network, and constructs forward and reverse propagation links independent of the backbone network. While maintaining the advantage of efficient parameters, it reduces the memory overhead during training and redundant calculations during reasoning multi-tasks, thereby achieving the purpose of reducing resource consumption during model training and improving computing efficiency.

[0205] The following is a further introduction to an efficient parameter and memory tuning method based on a pre-trained model proposed in an embodiment of the present invention.

[0206] Figure 8(c) is a schematic diagram of another tuning module according to an embodiment of the present invention. As shown in Figure 8(c), this embodiment separates the tuning module from the initial backbone network to form an independent sub-module. The separated sub-modules can act in parallel on the backbone network in the form of residuals and can be freely combined to form an independent initial bypass network.

[0207] It should be noted that the separated sub-modules can act on the backbone network in the form of residuals, or in other connection modes such as skip connections. No specific restrictions are made on the connection mode here.

[0208] Optionally, during the training process, the initial bypass network can use the intermediate output (feature data) of the backbone network during forward propagation, but during reverse propagation, it is disconnected from the backbone network, and the data flow only passes through the bypass network. Therefore, during reverse propagation, further calculation of the gradient of the backbone network can be avoided, thereby saving memory and improving training speed, thereby achieving the purpose of reducing resource consumption and improving computing efficiency during model training, and solving the problem of high resource consumption and low computing efficiency during model training.

[0209] As shown in Figure 8(c), the pre-trained model can be composed of a backbone network and an initial bypass network with residual connections, which is both parameter efficient and memory efficient. The backbone network can include multiple convolutional layers.

[0210] In this embodiment, the overall structure of the pre-trained model can be divided into two parts, one part is the backbone model, the parameters of the backbone model are frozen during the training process; the other part is the trainable initial bypass network (also called bypass structure), which can be composed of several tuning modules. The data flow from the tuning module to the backbone network can be cut off to form a new tuning module that is completely independent of the backbone network. Different tuning modules (Res-Tuner) can be combined to design and construct a complete initial bypass network. Based on the output results, the parameters of the initial bypass network can be updated to obtain the target bypass network.

[0211] In this embodiment, since the data flow between the initial bypass network and the backbone network is cut off during reverse propagation, the data flow of the tuning module in the initial bypass network does not flow to the backbone network, thereby saving memory and achieving the purpose of reducing memory loss.

[0212] Optionally, the output of the initial bypass network can be determined by the following formula:

[0213]

[0214]

[0215] in, It can be used to characterize the adjusted feature data corresponding to the first-level horizontal tuning module. It can be used to characterize the output of the bypass network of layer l, Res-Tuner(x l ) can be used to characterize the output of the level tuning module at layer l, It can be used to characterize the output of the vertical tuning module at the l-1th layer, x0 can be used to characterize the feature data output by the backbone network at the zeroth layer, and x l It can be used to characterize the output features of the lth layer, and λ can be used to characterize the weight of the initial bypass network.

[0216] Optionally, as shown in FIG8(c), each network layer corresponding to the backbone network has a corresponding horizontal tuning module and a vertical tuning module in the initial bypass network. The horizontal tuning module and the vertical tuning module can be used to process the output features from the lth layer of the backbone network and the feature outputs from the l-1th layer of the initial bypass network, respectively, and the outputs of the two can be weighted and summed using a learnable weight (λ coefficient).

[0217] Optionally, the weight coefficients may be continuously adjusted through a back-propagation algorithm.

[0218] For example, if the backbone network has 12 network layers, the feature data can be input into the initial bypass network after all the input data are converted by the 12 network layers. However, in order to improve the data processing efficiency, the feature data can be transmitted from the first network layer of the backbone network to the first stage of the initial bypass; and then the feature data can be transmitted from the second network layer of the backbone network to the second stage of the initial bypass, and so on.

[0219] As an optional embodiment, the pre-trained model can be applied to the discriminative task.

[0220] Fig. 9 is a schematic diagram of a pre-training model according to an embodiment of the present invention, such as Fig. 9 As shown, in the discriminative task, the overall structure of the pre-trained model can be divided into two parts. One part is the backbone network, which is frozen during the training process, wherein the backbone network may include multiple training layers (e.g., training layer_1, training layer_2...training layer_N), and the above training layers may be convolutional layers. The other part is a trainable initial bypass network (also referred to as a bypass structure), which may be composed of several tuning modules, wherein each network layer of the corresponding backbone network has a horizontal tuning module and a vertical tuning module, which are respectively used to receive the feature data output from the layer in the backbone network and the output of the previous layer in the initial bypass network, and the two are weighted summed and output to the next layer, until the converted feature data of the last layer is output to the output layer in the pre-trained model for the output of the corresponding result, and the back propagation process may be performed based on the output of the corresponding result, and in the back propagation process, the back propagation path from the bypass network to the backbone network will be disconnected to obtain the trained target bypass grid. When the image to be classified is obtained, the backbone model can be obtained to process the image to be classified to obtain the feature data. The target bypass network can be called to convert the feature data, and the converted feature data can be output to the output layer in the pre-trained model to determine the category to which the image to be classified belongs.

[0221] As an optional embodiment, the pre-trained model can be applied to the generative task.

[0222] Fig.10 is a schematic diagram of another pre-training model according to an embodiment of the present invention, such as Fig.10 As shown, in generative tasks (e.g., image generation), the overall structure of the pre-trained model may include a backbone network whose parameters are frozen during training and a target bypass network whose parameters are updated, wherein the backbone network may be a deep learning model (U-Net) structure based on stable diffusion (StableDiffusion), and may include a coding layer 64*64, a coding layer 32*32, a coding layer 16*16, a coding layer 8*8, an intermediate layer 8*8, a decoding layer 8*8, a decoding layer 16*16, a decoding layer 32*32, and a decoding layer 64*64. The target bypass network may be a trainable bypass structure, and may be composed of several tuning modules.

[0223] Each corresponding decoding layer of the backbone network corresponds to a horizontal tuning module and a vertical tuning module. The horizontal tuning module and the vertical tuning module can respectively receive feature data output from the middle layer or decoding layer in the backbone network and feature data output from the previous layer in the target bypass network, and can perform weighted summation on the two and output them to the vertical tuning module corresponding to the next layer of the backbone network, until the weighted summation result of the horizontal tuning module and the vertical tuning module corresponding to the last decoding layer of the backbone network is output to the generative task network for the output of the corresponding result to obtain the target data.

[0224] Optionally, the target bypass model can be used as the last encoding layer in the pre-trained model, and the converted feature data can be used as the output data of the last decoding layer in the pre-trained model. Based on the output data, the output layer can convert the output data to obtain the target data.

[0225] Optionally, the target bypass network in this embodiment can also be used as a branch in a controllable generative task (pre-training model), that is, it can be an encoding module for conditional input.

[0226] Controllable generative tasks are tasks that are generated based on controllable conditional data. In this embodiment, the controllable encoding part can also use the target bypass network as a branch to achieve the purpose of reducing memory consumption.

[0227] For example, the backbone network may include three encoding layers, one intermediate layer, and three decoding layers. Each decoding layer corresponds to a horizontal tuning module and a vertical tuning module, and the horizontal tuning module and the vertical tuning module can be used to receive feature data output from the intermediate layer or the decoding layer and feature data converted from the previous layer in the target bypass network. The first vertical tuning module corresponding to the first decoding layer can be used to receive feature data of the intermediate layer of the backbone network and convert the feature data of the intermediate layer. The first horizontal tuning module associated with the first vertical tuning module can obtain feature data output by the first decoding layer and convert feature data output by the first decoding layer. Based on the weight of the target bypass network, a weighted sum can be performed between the feature data adjusted by the first vertical tuning module and the feature data adjusted by the first horizontal tuning module.

[0228] Optionally, the feature data obtained by weighted summing the feature data adjusted by the first vertical tuning module and the feature data adjusted by the first horizontal tuning module can be input into the second vertical tuning module corresponding to the second decoding layer to obtain the feature data converted by the second vertical tuning module, and the feature data output by the second decoding layer is output to the second horizontal tuning module to obtain the corresponding feature data adjusted by the second horizontal tuning module, and the feature data adjusted by the second horizontal tuning module and the feature data adjusted by the second vertical tuning module can be weighted summed. The feature data obtained by weighted summing the feature data adjusted by the second horizontal tuning module and the feature data adjusted by the second vertical tuning module are continuously transmitted to the third vertical tuning module corresponding to the third decoding layer, and the feature data output by the third decoding layer is transmitted to the third horizontal tuning module corresponding to the third decoding layer, and the feature data adjusted by the third horizontal tuning module and the feature data adjusted by the third vertical tuning module can be weighted summed to obtain the feature data finally converted by the target bypass network. The feature data can be processed based on the output layer in the pre-trained model to obtain the target data.

[0229] In an embodiment of the present invention, a parameter- and memory-efficient initial bypass network is constructed with the help of a tuning module separated from the initial backbone, and the weights in the initial bypass network are updated to obtain a target bypass network. Compared with the original tuning method, this method can greatly reduce memory consumption and the cost of multi-task reasoning, and is applicable to both discriminative and generative tasks.

[0230] In an embodiment of the present invention, a tuning module extracted from a backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) that is completely independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby saving memory and improving the training speed, thereby achieving the technical effect of reducing resource consumption in the training process of the model and improving computing efficiency, and solving the technical problems of high resource consumption and computing efficiency in the training process of the model.

[0231] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention, such as weather forecast results and other data, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0232] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0233] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0234] Example 4

[0235] According to an embodiment of the present invention, there is also provided a method for implementing the above Figure 2 The method for updating parameters in the pre-trained model shown is a device for updating parameters in the pre-trained model.

[0236] Fig.11is a schematic diagram of a device for updating parameters in a pre-training model according to an embodiment of the present invention, such as Fig.11 As shown, the device 1100 for updating parameters in the pre-trained model may include: a first acquisition unit 1102 , a first calling unit 1104 and an updating unit 1106 .

[0237] The first acquisition unit 1102 is used to acquire feature data output by a backbone network of a pre-trained model, wherein the backbone network comes from an initial backbone network.

[0238] A first calling unit 1104 is used to call an initial bypass network to convert feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network;

[0239] The updating unit 1106 is used to update the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network, wherein in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network and the data flow of the backbone network are independent of each other, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data.

[0240] It should be noted that the first acquisition unit 1102, the first calling unit 1104 and the updating unit 1106 correspond to steps S202 to S206 in Example 1, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned units can also be run in the computer terminal provided in Example 1 as part of the device.

[0241] According to an embodiment of the present invention, another method for implementing the above Figure 3 The data processing device of the pre-trained model and the data processing method of the pre-trained model shown.

[0242] Fig.12 is a schematic diagram of a data processing device for a pre-training model according to an embodiment of the present invention. Fig.12 As shown, the data processing device 1200 of the pre-trained model may include: a first processing unit 1202, a second processing unit 1204, a first conversion unit 1206, a first generation unit 1208 and a display unit 1210.

[0243] The first processing unit 1202 is configured to respond to query information received in the generative interactive interface, wherein the query information at least includes: keywords of the text generation information.

[0244] The second processing unit 1204 is used to call the backbone network of the pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from the initial backbone network.

[0245] The first conversion unit 1206 is used to call the target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

[0246] A first generating unit 1208, configured to generate at least one reply result matching the query instruction based on the converted text feature data;

[0247] The display unit 1210 is used to display the reply result in the generative interactive interface.

[0248] It should be noted that the first processing unit 1202, the second processing unit 1204, the first conversion unit 1206, the first generation unit 1208 and the display unit 1210 correspond to steps S302 to S310 in Embodiment 1, and the five units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above units may be hardware components or software components stored in a memory and processed by one or more processors, and the above units may also be part of a device and run in a computer terminal provided in Embodiment 1.

[0249] According to an embodiment of the present invention, another method for implementing the above Figure 4 The data processing device of the pre-trained model and the data processing method of the pre-trained model shown.

[0250] Fig.13 is a schematic diagram of another data processing device for a pre-training model according to an embodiment of the present invention. Fig.13 As shown, the data processing device 1300 of the pre-training model may include: a second acquisition unit 1302, a second conversion unit 1304 and a second generation unit 1306.

[0251] The second acquisition unit 1302 is used to obtain feature data obtained by processing the conditional data by the backbone network in the pre-trained model, wherein the backbone network comes from the initial backbone network, and the conditional data is used to determine the generation condition of the target data.

[0252] The second conversion unit 1304 is used to call a target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network;

[0253] The second generating unit 1306 is used to generate target data corresponding to the condition data based on the converted feature data, wherein the type of the target data includes at least one of the following: text information, image information, video information and voice information.

[0254] It should be noted that the second acquisition unit 1302, the second conversion unit 1304 and the second generation unit 1306 correspond to steps S402 to S406 in Embodiment 1, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above units may be hardware components or software components stored in a memory and processed by one or more processors, and the above units may also be part of the device and run in the computer terminal provided in Embodiment 1.

[0255] According to an embodiment of the present invention, another method for implementing the above Figure 5 The data processing device of the pre-trained model and the data processing method of the pre-trained model shown.

[0256] Fig.14 is a schematic diagram of another data processing device for a pre-training model according to an embodiment of the present invention. Fig.14 As shown, the data processing device 1400 of the pre-trained model may include: an input unit 1402, a third processing unit 1404, a third conversion unit 1406 and a third generation unit 1408.

[0257] The input unit 1402 is used to input multimodal information in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: query information including character information, video frame information including frame image information, and audio information.

[0258] The third processing unit 1404 is used to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network.

[0259] The third conversion unit 1406 is used to call the target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

[0260] The third generating unit 1408 is used to generate reply information corresponding to the multimodal information based on the converted feature data, wherein the type of the reply information includes at least one of the following: text information, image information, video information and voice information.

[0261] It should be noted that the above input unit 1402, the third processing unit 1404, the third conversion unit 1406 and the third generation unit 1408 correspond to steps S702 to S708 in Embodiment 1, and the four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above units can be hardware components or software components stored in a memory and processed by one or more processors, and the above units can also be run in the computer terminal provided in Embodiment 1 as part of the device.

[0262] In the above-mentioned device, a tuning module extracted from the backbone network is used to construct an initial bypass network. At the same time, during the adjustment of the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) that is completely independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby saving memory and improving the training speed, thereby reducing the resource consumption in the training process of the model and solving the technical problems of high resource consumption and computing efficiency in the training process of the model.

[0263] Example 5

[0264] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0265] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0266] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for updating parameters in the pre-trained model: obtaining feature data output by the backbone network of the pre-trained model, wherein the backbone network comes from the initial backbone network; calling the initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; updating the parameters of the initial bypass network based on the converted feature data to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data.

[0267] Optionally, Fig.15 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Fig.15 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1502 , a memory 1504 and a transmission device 1506 .

[0268] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for updating parameters in the pre-trained model in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the method for updating parameters in the pre-trained model mentioned above is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0269] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the feature data output by the backbone network of the pre-trained model, wherein the backbone network comes from the initial backbone network; call the initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; update the parameters of the initial bypass network based on the converted feature data to obtain the target bypass network, wherein in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data.

[0270] Optionally, the processor may also execute program codes of the following steps: calling a tuning module in the initial bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the initial bypass network.

[0271] Optionally, the processor may also execute program codes of the following steps: calling a tuning module in the initial bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the initial bypass network.

[0272] Optionally, the processor may also execute the following program code: based on the weight of the initial bypass network, performing weighted summation on the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module to obtain the converted first feature data.

[0273] Optionally, the processor may also execute program code of the following steps: in response to the first-layer backbone network being the last layer of the backbone network, updating parameters of the initial bypass network based on the first characteristic data to obtain a target bypass network.

[0274] Optionally, the processor may also execute the following steps of program code: in response to the last layer of network in the non-backbone network of the first-layer backbone network, call the second vertical tuning module corresponding to the second-layer backbone network in the backbone network to adjust the first feature data, and call the second horizontal tuning module associated with the second vertical tuning module to adjust the feature data of the second-layer backbone network in the backbone network; based on the weight of the initial bypass network, perform weighted summation on the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the converted second feature data; in response to the last layer of network in the non-backbone network of the second-layer backbone network, perform the following steps: confirm the second-layer backbone network The method comprises the following steps: determining the first layer backbone network as the first layer backbone network, and determining the third layer backbone network in the backbone network as the second layer backbone network; calling the second vertical tuning module to adjust the first feature data, and calling the second horizontal tuning module to adjust the feature data of the second layer backbone network; based on the weight of the initial bypass network, performing weighted summation between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the converted second feature data, until the second layer backbone network is the last layer network in the backbone network; updating the parameters of the initial bypass network based on the converted feature data to obtain the target bypass network, including: updating the parameters of the initial bypass network based on the second feature data to obtain the target bypass network.

[0275] Optionally, the processor may also execute the program code of the following steps: determining a processing result obtained by processing the converted feature data by the output layer in the pre-trained model; determining a difference value between the processing result and a true processing result corresponding to the processing result; and adjusting the parameters of the initial bypass network based on the difference value to obtain a target bypass network.

[0276] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: respond to the query information received in the generative interactive interface, wherein the query information at least includes: keywords of the text generation information; call the backbone network of the pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from the initial backbone network; call the target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted text feature data, generate at least one reply result that matches the query instruction; and display the reply result in the generative interactive interface.

[0277] Optionally, the processor may also execute program code of the following steps: in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data.

[0278] As another optional example, the processor may call the information and application stored in the memory through the transmission device to perform the following steps: obtaining feature data obtained by processing the conditional data by the backbone network in the pre-trained model, wherein the backbone network comes from the initial backbone network, and the conditional data is used to determine the generation conditions of the target data; calling the target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generating target data corresponding to the conditional data, wherein the type of the target data includes at least one of the following: text information, image information, video information and voice information.

[0279] Optionally, the processor may also execute the program code of the following steps: calling the first vertical tuning module corresponding to the first decoding layer in the backbone network to convert the text feature data output by the middle layer of the backbone network, and calling the first horizontal tuning module associated with the first vertical tuning module to convert the text feature data output by the first decoding layer.

[0280] Optionally, the processor may also execute the program code of the following steps: based on the weight of the target bypass network, performing weighted summation on the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain converted text feature data; and determining a response result based on the converted text feature data.

[0281] Optionally, the processor may also execute the following program code: using the converted text feature data as output data of the last decoding layer in the pre-trained model; and converting the output data into a response result.

[0282] As another optional example, the processor may call the information and application stored in the memory through the transmission device to perform the following steps: input multimodal information in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: text information containing character information, video frame information containing frame image information, and audio information; call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network; call the target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generate reply information corresponding to the multimodal information, wherein the type of the reply information includes at least one of the following: text information, image information, video information, and voice information.

[0283] Optionally, the processor may also execute program codes of the following steps: calling a tuning module in the target bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the target bypass network.

[0284] Optionally, the processor may also execute the following program code: processing the converted feature data based on the output layer in the pre-trained model to obtain response information corresponding to the multimodal information.

[0285] By adopting the embodiment of the present invention, a tuning module extracted from the backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is truncated to obtain a target bypass network (new tuning module) that is completely independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby achieving memory savings and improved training speed, thereby achieving the technical effect of reducing resource consumption during the training process of the model, and solving the technical problems of high resource consumption and computing efficiency during the training process of the model.

[0286] It can be understood by those skilled in the art that Fig.15 The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.15It does not limit the structure of the above-mentioned computer terminal A. For example, the computer terminal A may also include Fig.15 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.15 Different configurations are shown.

[0287] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0288] Example 6

[0289] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method for updating parameters in the pre-trained model provided in the first embodiment.

[0290] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0291] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the information stored in the memory and the application program executed by the processor through the transmission device.

[0292] In an embodiment of the present invention, a tuning module extracted from a backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) that is completely independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby achieving memory savings and improved training speed, thereby achieving the technical effect of reducing resource consumption during the training process of the model, and solving the technical problems of high resource consumption and computational efficiency during the training process of the model.

[0293] Example 7

[0294] An embodiment of the present invention may provide an electronic device, which may include a memory and a processor.

[0295] Fig.16It is a block diagram of an electronic device for a method of updating parameters in a pre-trained model according to an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0296] like Fig.16 As shown, the device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 to a random access memory (RAM) 1603. In RAM 1603, various programs and data required for the operation of the device 1600 can also be stored. The computing unit 1601, ROM 1602, and RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0297] A number of components in the device 1600 are connected to the I / O interface 1605, including: an input unit 1606, such as a keyboard, a mouse, etc.; an output unit 1604, such as various types of displays, speakers, etc.; a storage unit 1608, such as a disk, an optical disk, etc.; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1609 allows the device 1600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0298] The computing unit 1601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1601 performs the various methods and processes described above, such as a method for updating parameters in a pre-trained model. For example, in some embodiments, the method for updating parameters in a pre-trained model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1600 via the ROM 1602 and / or the communication unit 16010. When the computer program is loaded into RAM 1603 and executed by the computing unit 1601, one or more steps of the method for updating parameters in the pre-trained model described above may be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to execute the method for updating parameters in the pre-trained model in any other appropriate manner (e.g., by means of firmware).

[0299] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0300] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0301] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0302] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a cathode ray tube or liquid crystal display, monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or path ball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0303] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks, wide area networks, and the Internet.

[0304] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0305] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0306] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0307] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0308] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0309] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0310] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memory random access memories, mobile hard disks, magnetic disks or optical disks.

[0311] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for updating parameters in a pre-trained model, characterized in that: include: Acquire feature data output by a backbone network of a pre-trained model, wherein the backbone network comes from an initial backbone network; Calling an initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial trunk network; Based on the converted characteristic data, the parameters of the initial bypass network are updated to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the trunk network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data.

2. The method according to claim 1, characterized in that Calling the initial bypass network to convert the characteristic data includes: Calling the tuning module in the initial bypass network to adjust the characteristic data; Based on the weight of the initial bypass network, weighted summation is performed on the adjusted feature data.

3. The method according to claim 2, characterized in that Calling the tuning module in the initial bypass network to adjust the characteristic data includes: The first vertical tuning module corresponding to the first layer of the backbone network in the backbone network is called to adjust the characteristic data output by the zeroth layer of the backbone network in the backbone network, and the first horizontal tuning module associated with the first vertical tuning module is called to adjust the characteristic data output by the first layer of the backbone network.

4. The method according to claim 3, characterized in that Based on the weight of the initial bypass network, weighted summing is performed on the adjusted characteristic data, including: Based on the weight of the initial bypass network, a weighted sum is performed on the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module to obtain converted first feature data.

5. The method according to claim 4, characterized in that Updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: In response to the first layer of backbone network being the last layer of network in the backbone network, the parameters of the initial bypass network are updated based on the first characteristic data to obtain the target bypass network.

6. The method according to claim 4, characterized in that Updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: In response to the first layer of backbone network not being the last layer of network in the backbone network, calling a second vertical tuning module corresponding to a second layer of backbone network in the backbone network to adjust the first feature data, and calling a second horizontal tuning module associated with the second vertical tuning module to adjust the feature data of the second layer of backbone network in the backbone network; Based on the weight of the initial bypass network, performing a weighted summation between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain converted second feature data; In response to the second layer backbone network being not the last layer network in the backbone network, performing the following steps: Determine the second-layer backbone network as the first-layer backbone network, and determine the third-layer backbone network in the backbone network as the second-layer backbone network; Calling the second vertical tuning module to adjust the first characteristic data, and calling the second horizontal tuning module to adjust the characteristic data of the second layer backbone network; Based on the weight of the initial bypass network, weighted sum is performed between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain converted second feature data, until the second-layer backbone network is the last layer of the backbone network; Updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: updating the parameters of the initial bypass network based on the second characteristic data to obtain the target bypass network.

7. The method according to claim 1, characterized in that Updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: Determine a processing result obtained by processing the converted feature data by an output layer in the pre-trained model; Determine a difference value between the processing result and a true processing result corresponding to the processing result; The parameters of the initial bypass network are adjusted based on the difference value to obtain the target bypass network.

8. The method according to claim 1, characterized in that The backbone network and the at least one tuning module are connected based on the residual.

9. The method according to claim 1, characterized in that: During the process of transmitting the characteristic data to the initial bypass network, the state of the parameters of the backbone network is a locked state.

10. A method for processing data in a pre-training model, characterized in that: include: Respond to the query information received in the generative interactive interface, wherein the query information at least includes: Keywords for text-generated information; Calling a backbone network of a pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from an initial backbone network; Calling a target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating parameters of an initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; Based on the converted text feature data, generating at least one reply result matching the query instruction; The response result is displayed in the generative interactive interface.

11. The method according to claim 10, characterized in that in, In the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the text feature data.

12. The method according to claim 11, characterized in that Calling the target bypass network to convert the text feature data includes: The first vertical tuning module corresponding to the first decoding layer in the backbone network is called to convert the text feature data output by the middle layer of the backbone network, and the first horizontal tuning module associated with the first vertical tuning module is called to convert the text feature data output by the first decoding layer.

13. The method according to claim 12, characterized in that Based on the converted text feature data, generating at least one reply result matching the query instruction, including: Based on the weight of the target bypass network, performing a weighted summation between the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain converted text feature data; The reply result is determined based on the converted text feature data.

14. The method according to claim 11, characterized in that Determining the reply result based on the converted text feature data includes: Using the converted text feature data as output data of the last decoding layer in the pre-training model; The output data is converted into the response result.

15. A data processing method for a pre-training model, characterized in that: include: Inputting multimodal information in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: text information including character information, video frame information including frame image information, and audio information; Calling a backbone network in a pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from an initial backbone network; Calling a target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating parameters of an initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; Based on the converted feature data, reply information corresponding to the multimodal information is generated, wherein the type of the reply information includes at least one of the following: text information, image information, video information and voice information.

16. The method according to claim 15, characterized in that Calling the target bypass network to convert the characteristic data includes: Calling the tuning module in the target bypass network to adjust the characteristic data; Based on the weight of the target bypass network, a weighted sum is performed on the adjusted characteristic data.

17. The method according to claim 15, characterized in that Generating reply information corresponding to the multimodal information based on the converted feature data, including: The converted feature data is processed based on the output layer in the pre-trained model to obtain the response information corresponding to the multimodal information.

18. A data processing system for a pre-trained model, characterized in that: include: A front-end client, used to display a dialogue interface and capture multimodal information input in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: query information containing character information, video frame information containing frame image information, and audio information; A backend server is used to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and generate reply information corresponding to the multimodal information based on the converted feature data, wherein the backbone network comes from the initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of the reply information includes at least one of the following: text information, image information, video information and voice information.

19. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 17 are implemented.

Images