Information processing method and apparatus, device, and storage medium

Abstract

Embodiments of the present application relate to an information processing method and apparatus, a device, and a storage medium. The method provided herein comprises: providing input information to a target model, the target model comprising a plurality of processing layers, and the plurality of processing layers at least including a first processing layer and a second processing layer adjacent to each other; on the basis of the input information, determining a first input feature associated with the first processing layer, the first input feature comprising a first set of feature components; applying a first set of weight parameters to the first set of feature components to determine an intermediate input feature; determining an intermediate output feature generated by the first processing layer on the basis of the intermediate input feature; on the basis of the intermediate output feature and the first set of feature components, determining a second input feature associated with the second processing layer; and generating an output result of the target model on the basis of at least the second input feature.

Description

Method, device, equipment and storage medium for information processing TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, device, equipment and computer-readable storage medium for information processing. BACKGROUND

[0002] With the development of computer technology, various types of machine models are gradually applied to various scenarios. For example, a generative model based on a transformer can be used to perform various types of generation tasks. In the architecture of a neural network model, residual connection is an important design that helps to avoid the occurrence of gradient disappearance problem.

[0003] SUMMARY

[0004] In a first aspect of the present disclosure, a method for information processing is provided. The method comprises: providing input information to a target model, the target model comprising a plurality of processing layers, the plurality of processing layers comprising at least a first processing layer and a second processing layer adjacent to each other; determining, based on the input information, a first input feature associated with the first processing layer, the first input feature comprising a first set of feature components; applying a first set of weight parameters to the first set of feature components to determine an intermediate input feature; determining an intermediate output feature generated by the first processing layer based on the intermediate input feature; determining, based on the intermediate output feature and the first set of feature components, a second input feature associated with the second processing layer; and generating an output result of the target model based at least on the second input feature.

[0005] In a second aspect of the present disclosure, an apparatus for information processing is provided. The apparatus comprises: an input providing module configured to provide input information to a target model, the target model comprising a plurality of processing layers, the plurality of processing layers comprising at least a first processing layer and a second processing layer adjacent to each other; a feature determining module configured to determine, based on the input information, a first input feature associated with the first processing layer, the first input feature comprising a first set of feature components; a first weighting module configured to apply a first set of weight parameters to the first set of feature components to determine an intermediate input feature; a feature processing module configured to determine an intermediate output feature generated by the first processing layer based on the intermediate input feature; a second weighting module configured to determine, based on the intermediate output feature and the first set of feature components, a second input feature associated with the second processing layer; and a result output module configured to generate an output result of the target model based at least on the second input feature.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon a computer program, the computer program being executable by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product includes computer executable instructions that, when executed by a processor, implement the method according to the first aspect of the present disclosure.

[0009] It should be understood that nothing in the Summary is to be construed as a limitation of the key or important features of the embodiments of the disclosure or as limiting the scope of the disclosure. Other features of the disclosure will be more fully understood from the following description of the embodiments of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects and advantages of embodiments of the present disclosure will become more apparent from the following detailed description of the embodiments when taken in conjunction with the accompanying drawings. In the drawings:

[0011] FIG. 1 shows a schematic diagram of a conventional residual connection;

[0012] FIG. 2 shows a schematic diagram of an example process of information processing according to some embodiments of the present disclosure;

[0013] FIG. 3 shows an example connection architecture according to some embodiments of the present disclosure;

[0014] FIG. 4 shows comparative test results according to some embodiments of the present disclosure;

[0015] FIG. 5 shows a schematic structural block diagram of an example apparatus for information processing according to some embodiments of the present disclosure; and

[0016] FIG. 6 shows a block diagram of an electronic device capable of implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should be noted that the titles of any sections / sub-sections provided herein are not limiting. Various embodiments are described throughout this document and any type of embodiment can be included under any section / sub-section. Furthermore, embodiments described in any section / sub-section can be combined with any other embodiments described in the same section / sub-section and / or different section / sub-section in any manner.

[0019] In the description of embodiments of the present disclosure, the term "includes" and its derivatives, such as "including," should be understood in an open, inclusive sense, that is, "including, but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "an embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicitly and implicitly recited definitions can also be found below. The terms "first," "second," and the like can refer to different or the same objects. Other explicit and implicit definitions can also be found below.

[0020] Data of users, acquisition and / or use of data, etc. can be involved in embodiments of the present disclosure. These aspects all comply with corresponding laws and regulations and relevant provisions. In embodiments of the present disclosure, all collection, acquisition, processing, processing, forwarding, use, etc. of data are performed on the premise that users are aware of and confirm. Accordingly, when implementing embodiments of the present disclosure, the type of data or information that can be involved, the range of use, the scenario of use, etc. should be informed to users and authorized by users in a proper manner according to relevant laws and regulations. The specific informing and / or authorization manner can vary according to actual situations and application scenarios, and the scope of the present disclosure is not limited in this respect.

[0021] In the present specification and embodiments, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. Users refuse to process personal information other than the necessary information required for basic functions, which will not affect the user's use of basic functions.

[0022] In the architecture of a neural network model, residual connection is an important design that helps to avoid the occurrence of gradient vanishing problem. FIG. 1 shows a schematic diagram of a conventional residual connection. As shown in the figure, after performing F(X) on the input feature X using a processing layer, the input feature X can be added to the output feature F(X) as the input feature of the next processing layer. However, as the hidden features of the deeper layers of the network become highly similar, this reduces the contribution of the additional layers and can lead to representation collapse.

[0023] Embodiments of the present disclosure propose a scheme of information processing. The scheme includes: providing input information to a target model, the target model including a plurality of processing layers, the plurality of processing layers including at least a first processing layer and a second processing layer adjacent to each other; determining, based on the input information, a first input feature associated with the first processing layer, the first input feature including a first set of feature components; applying a first set of weight parameters to the first set of feature components to determine an intermediate input feature; determining an intermediate output feature generated by the first processing layer based on the intermediate input feature; determining, based on the intermediate output feature and the first set of feature components, a second input feature associated with the second processing layer; and generating, based at least on the second input feature, an output result of the target model.

[0024] In this way, embodiments of the present disclosure can introduce learnable connection weights, providing a more powerful and flexible connection mechanism for the model, helping to solve the representation collapse caused by residual connection, thereby improving the efficiency of model processing.

[0025] Various example implementations of the scheme are described in further detail below in conjunction with the accompanying drawings.

[0026] FIG. 2 shows a flowchart of an example information processing process 200 according to some embodiments of the present disclosure. The process 200 can be implemented at a suitable electronic device, which can be deployed with a target model.

[0027] As shown in FIG. 2, at block 210, the electronic device provides input information to a target model, the target model including a plurality of processing layers, the plurality of processing layers including at least a first processing layer and a second processing layer adjacent to each other.

[0028] FIG. 3 shows an example connection architecture 300 according to some embodiments of the present disclosure. As shown in FIG. 3, the architecture of the target model can include a plurality of processing layers, e.g., processing layer 315. In some embodiments, the target model can be constructed by replacing a set of processing layers connected by residual connection in a transformer unit with a plurality of processing layers connected by the connection manner introduced below.

[0029] In some embodiments, the target model can also be a generative model, and the input information received by the target model can include textual content. The textual content can be input as a prompt for the generative model.

[0030] Such a generative model can be used to perform a suitable type of generative task, e.g., generating textual content, image content (picture or video), audio content, etc. based on an input prompt.

[0031] At block 220, the electronic device determines, based on the input information, a first input feature associated with the first processing layer, the first input feature including a first set of feature components.

[0032] Taking FIG. 3 as an example, the electronic device 110 can utilize the target model to process the input information, and can determine the first input feature 310 of the processing layer 315. As shown in FIG. 3, the first input feature 310 can include a plurality of feature components, e.g., feature component 305-1 and feature component 305-2.

[0033] As an example, h k-1 may be used to represent the input feature associated with the kth layer. The initial input h 0 may be replicated n times to constitute the initial hidden feature matrix Accordingly, the input feature matrix of the kth processing layer can be represented as wherein represents the i-th feature component in the input feature of the kth processing layer.

[0034] At block 230, the electronic device applies the first set of weight parameters to the first set of feature components to determine an intermediate input feature.

[0035] As shown in FIG. 3, taking the first input feature including feature component 305-1 and feature component 305-2 as an example, the input feature of the processing layer 315 can be determined based on a weighted sum of feature component 305-1 and feature component 305-2. As an example, the intermediate input feature can be represented as: a 1，0 *h1+ a 2，0 *h2, wherein a 1，0 and a 2，0 are the first set of weight parameters.

[0036] As an example, the first input feature can be represented as H = (h1 h2... h n ) T , and the intermediate input feature of the processing layer can be represented as: wherein A m represents the first set of weight parameters (a 1，0 , a 2，0,... a n，0 ).

[0037] With reference back to FIG. 2, at block 240, the electronic device determines intermediate output features generated by the first processing layer based on the intermediate input features.

[0038] As an example, the processing layer 315 can perform processing (e.g., the process can be represented as ) on the intermediate input features, and the corresponding intermediate input features can be represented as

[0039] At block 250, the electronic device determines the second input features associated with the second processing layer based on the intermediate output features and the first set of feature components.

[0040] With reference back to FIG. 3, the electronic device can determine the second set of feature components of the second input features based on the intermediate output features and the first set of feature components H = (h1 h2... h n ) T , where each feature component of the second set of feature components is determined by applying the corresponding second set of weight parameters to the intermediate output features and the first set of feature components.

[0041] As an example with reference to FIG. 3, the first feature component of the second output features can be represented as:

[0042] The second feature component of the second output features can be represented as:

[0043] Thus, embodiments of the present disclosure consider both the width connection of different feature components of the first input features and the depth connection between the first input features and the intermediate output features of the processing layer.

[0044] The determination process of the second input features can also be represented as a matrix operation, which can process the first input features based on a matrix of weight parameters to determine the second input features. As an example, can be represented as:

[0045] The processing process of the second input features can be represented as:

[0046] In the above parameter matrix, the weight parameters related to the width connection can be represented as:

[0047] In the above parameter matrix, the weight parameters related to the depth connection can be represented as:

[0048] In some embodiments, the matrix of weight parameters The included weight parameters can be static parameters determined by training the target model. Taking FIG. 3 as an example, the first set of weight parameters and / or the second set of weight parameters, for example, β1, β2, α 1，0 , α 2，0 , α 1，1 , α 1，2 , α 2，1 , α 2，2 , can be static model parameters determined by training the target model.

[0049] In some embodiments, the first set of weight parameters and / or the second set of weight parameters can also be a set of dynamic parameters determined by the first input feature.

[0050] As an example, the matrix of weight parameters can be represented as:

[0051] Correspondingly, the process of determining the second input feature based on the first input feature can be represented as:

[0052] In some embodiments, the electronic device can dynamically determine the weight parameters based on the following process:

[0053] Specifically, as shown in equation (9), the electronic device can normalize the first input feature H to determine the reference feature

[0054] Further, as represented by equations (10) to (12), the electronic device can determine the second set of static parameters determined by training the target model. Specifically, the electronic device can determine the static parameters s β , s α , s γ , W β , W m , W γ , where B, A m and A γ are preset parameters.

[0055] Further, the electronic device can perform linear transformation on the reference feature according to equations (10) to (12). Specifically, the electronic device can perform linear transformation on the reference feature based on the second set of static parameters to determine the first set of weight parameters and / or the second set of weight parameters, thereby determining the dynamic matrix of weight parameters

[0056] By using dynamic weight parameters to realize feature transmission, embodiments of the present disclosure can autonomously adjust the connection strength between network layers through learning algorithms to optimize feature representations at different depths. This adaptive method significantly reduces the representation collapse problem commonly seen in deep networks, enhances the representation capacity of the model, and improves the stability and convergence speed of the network by improving gradient flow. In addition, it improves the utilization efficiency of computing resources by almost not increasing additional parameters and computational overhead, while providing greater flexibility and adaptability, enabling the network to better adapt to various input features and task requirements. On the other hand, this connection method also simplifies the network design process, as the network can automatically learn the optimal connection method, reducing the need for human-designed and adjusted network architecture. Based on the above, embodiments of the present disclosure can bring significant performance improvement and computational efficiency to the model, while reducing the burden on designers in network architecture design.

[0057] With continued reference to FIG. 2, at block 260, the electronic device generates an output result of the target model based at least on the second input features.

[0058] As an example, the second processing layer can process the second input features based on the architecture described with reference to FIG. 3, and consider the depth connection and width connection to determine the input features of the next processing layer.

[0059] Further, the target model can determine the final output result of the model through the processing layer connection architecture described above. As mentioned above, the target model can be a generative model, accordingly, its input result can include but is not limited to: text content, image content (picture or video), audio content, etc.

[0060] Based on the connection mechanism described above, embodiments of the present disclosure allow the neural network to dynamically learn and adjust the connection strength between different levels, thereby more effectively solving the representation collapse and gradient vanishing problems inherent in traditional residual connections. This adaptive connection method not only improves the representation capacity of the network, enabling each layer to learn more rich and discriminative features, but also enhances the stability and convergence speed of the network during the training process by optimizing the gradient flow. In addition, embodiments of the present disclosure can significantly improve the performance of the model, including language modeling, image classification, and diffusion models, demonstrating its wide applicability and effectiveness.

[0061] FIG. 4 further illustrates comparative test results according to some embodiments of the present disclosure. As shown in FIG. 4, for the conventional residual connection, the cosine similarity gradually increases as the number of layers increases, which indicates that the features gradually tend to be consistent as they propagate through the network, which can lead to collapse of the representation, i.e., different layers of the network learn increasingly similar representations. In contrast, the cosine similarity for the connection proposed in the present disclosure is significantly lower than the residual connection and the range of similarity is wider, which indicates that the connection proposed in the present disclosure can reduce the similarity of the representations between adjacent layers, thereby helping each layer of the network to learn more diverse and discriminative features.

[0062] Example apparatus and device

[0063] Embodiments of the present disclosure also provide a corresponding apparatus for implementing the above-mentioned method or process. FIG. 5 shows a schematic structural block diagram of an example apparatus 500 for information processing according to certain embodiments of the present disclosure. The apparatus 500 can be implemented as or included in an electronic device as discussed above. Various modules / components in the apparatus 500 can be implemented by hardware, software, firmware, or any combination thereof.

[0064] As shown in FIG. 5, the apparatus 500 includes an input providing module 510 configured to provide input information to a target model, the target model including a plurality of processing layers, the plurality of processing layers including at least a first processing layer and a second processing layer adjacent to each other; a feature determining module 520 configured to determine, based on the input information, a first input feature associated with the first processing layer, the first input feature including a first set of feature components; a first weighting module 530 configured to apply a first set of weight parameters to the first set of feature components to determine an intermediate input feature; a feature processing module 540 configured to determine an intermediate output feature generated by the first processing layer based on the intermediate input feature; a second weighting module 550 configured to determine, based on the intermediate output feature and the first set of feature components, a second input feature associated with the second processing layer; and a result output module 560 configured to generate an output result of the target model based at least on the second input feature.

[0065] In some embodiments, the second weighting module 550 is further configured to determine, based on the intermediate output feature and the first set of feature components, a second set of feature components of the second input feature, wherein each feature component of the second set of feature components is determined by applying a corresponding second set of weight parameters to the intermediate output feature and the first set of feature components.

[0066] In some embodiments, the first set of weight parameters and / or the second set of weight parameters are a first set of static parameters determined by training the target model.

[0067] In some embodiments, the first set of weight parameters and / or the second set of weight parameters are a set of dynamic parameters determined by the first input features.

[0068] In some embodiments, the first set of weight parameters and / or the second set of weight parameters are determined based on a process of: normalizing the first input features to determine reference features; determining a second set of static parameters determined by training the target model; and performing a linear transformation on the reference features based on the second set of static parameters to determine the first set of weight parameters and / or the second set of weight parameters.

[0069] In some embodiments, the target model is constructed by replacing a set of processing layers connected by residual connections in the transformer unit with a plurality of processing layers.

[0070] In some embodiments, the target model is a generative model, and the input information includes at least text content, the text content being provided as a prompt word for the generative model, and the output result includes one of: text content, image content, audio content.

[0071] The modules included in the apparatus 500 can be implemented utilizing a variety of means, including software, hardware, firmware, or any combination of these. In some embodiments, one or more units can be implemented using software and / or firmware, e.g., machine executable instructions stored on a machine readable medium. In addition to or alternatively, some or all of the modules in the apparatus 500 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0072] FIG. 6 illustrates a block diagram of an electronic device 600 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 600 illustrated in FIG. 6 is merely exemplary and should not be construed as limiting on the functionality and scope of the embodiments described herein. The electronic device 600 illustrated in FIG. 6 can be used to implement an electronic device as discussed above.

[0073] As shown in FIG. 6, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 can include, but are not limited to, one or more processors or processing units 610, memory 620, storage 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit(s) 610 can be actual or virtual processors and capable of executing various processing in accordance with programs stored in memory 620. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel to improve the processing power of electronic device 600.

[0074] Electronic device 600 typically includes a plurality of computer storage media. Such media can be removable and / or non-removable, and can include volatile and / or nonvolatile media. Memory 620 can be volatile (such as, for example, registers, cache, random access memory (RAM)), non-volatile (such as, for example, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage 630 can be removable or non-removable and can include machine-readable media, such as, for example, flash drives, disks, or any other media capable of storing information and / or data and accessible by electronic device 600.

[0075] Electronic device 600 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 6, a disk drive or other computer-readable media drive can be provided for reading from or writing to a removable, non- volatile magnetic disk (e.g., a "hard drive"), and a disk drive or other computer-readable media drive can be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM or other optical medium). In these instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. Memory 620 can include a computer program product 625 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0076] Communication unit(s) 640 enable communication with other electronic devices via communication media. Additionally, functionality of components of electronic device 600 can be implemented in a single computing cluster or a plurality of computer machines capable of communication through a communication connection. Accordingly, electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes.

[0077] The input device 650 can be one or more input devices such as a mouse, a keyboard, a trackball, etc. The output device 660 can be one or more output devices such as a display, a speaker, a printer, etc. The electronic device 600 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc., one or more devices that enable a user to interact with the electronic device 600, or any devices (e.g., a network card, a modem, etc.) that enable the electronic device 600 to communicate with one or more other electronic devices, as desired, via the communication unit 640. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0078] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0079] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0080] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0081] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0082] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (media) having instances of the software embodied thereon, such as computer software, firmware, wireless application protocol (WAP), middleware or microcode. For example, a computer program product can be a floppy disk, a CD-ROM, a DVD, a Blu-ray Disc™, a flash drive, a memory stick, a magnetic tape, or a hard disk drive. The machine-readable medium can be a single medium, or multiple media, of the same or different type. The computer program product can be one or more computer program components embodied in medium and / or transmission signals. The computer program product can have one or more computer program components embodied in medium and / or transmission signals.

[0083] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although the implementations of the disclosure have been described with regard to one or more implementations, it will be recognized that a variety of modifications and changes can be made to these implementations without departing from the broader spirit and scope of the implementations as set forth in the preceding disclosure. For example, certain aspects of the implementations can be performed using hardware, software, and / or firmware, or any combination thereof. The above-described implementations should therefore be regarded as merely illustrative, and not as narrowing the scope of the disclosure, which is defined by the appended claims and their equivalents.

Claims

A method of information processing, comprising: providing input information to a target model, the target model comprising a plurality of processing layers, the plurality of processing layers comprising at least an adjacent first processing layer and a second processing layer; determining, based on the input information, a first input feature associated with the first processing layer, the first input feature comprising a first set of feature components; applying a first set of weight parameters to the first set of feature components to determine an intermediate input feature; determining an intermediate output feature generated by the first processing layer based on the intermediate input feature; determining, based on the intermediate output feature and the first set of feature components, a second input feature associated with the second processing layer; and generating, based on at least the second input feature, an output result of the target model. The method of claim 1, wherein determining, based on the intermediate output feature and the first set of feature components, a second input feature associated with the second processing layer comprises: determining, based on the intermediate output feature and the first set of feature components, a second set of feature components of the second input feature, wherein each feature component of the second set of feature components is determined by applying a corresponding second set of weight parameters to the intermediate output feature and the first set of feature components. The method of claim 2, wherein the first set of weight parameters and / or the second set of weight parameters are a first set of static parameters determined by training the target model. The method of claim 2, wherein the first set of weight parameters and / or the second set of weight parameters are a set of dynamic parameters determined by the first input feature. The method of claim 4, wherein the first set of weight parameters and / or the second set of weight parameters are determined based on: normalizing the first input feature to determine a reference feature; determining a second set of static parameters determined by training the target model; and performing a linear transformation on the reference feature based on the second set of static parameters to determine the first set of weight parameters and / or the second set of weight parameters. The method of claim 1, wherein the target model is constructed by replacing a set of processing layers connected by residual connections in a transformer unit with the plurality of processing layers. The method of claim 1, wherein the target model is a generative model, and the input information comprises at least textual content provided as a prompt word for the generative model, and the output result comprises one of: textual content, image content, audio content. An apparatus for information processing, comprising: an input providing module configured to provide input information to a target model, the target model comprising a plurality of processing layers, the plurality of processing layers comprising at least an adjacent first processing layer and a second processing layer; a feature determining module configured to determine, based on the input information, a first input feature associated with the first processing layer, the first input feature comprising a first set of feature components; a first weighting module configured to apply a first set of weight parameters to the first set of feature components to determine an intermediate input feature; a processing determining module configured to determine an intermediate output feature generated by the first processing layer based on the intermediate input feature; ​ ​ a feature processing module, configured to determine an intermediate output feature generated by the first processing layer based on the intermediate input feature; a second weighting module, configured to determine a second input feature associated with the second processing layer based on the intermediate output feature and the first set of feature components; and a result output module, configured to generate an output result of the target model based on at least the second input feature. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-8. A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-8. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1-8.

Images