Information processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an information processing method and device, equipment and a storage medium. The method comprises the steps of obtaining input content of a model; determining, using an attention unit of the model, a first query feature and a first key feature based on the input content; processing the first query feature and the first key feature using a dimensionality reduction unit of the model to determine a second query feature and a second key feature; determining attention information based on the second query feature and the second key feature; and generating response content of the model for the input content based on the attention information. Based on the mode, the attention can be accurately estimated by reducing the dimensions of the query features and the key features, the accuracy of information processing is guaranteed, and the calculation overhead and the memory overhead of the model in the reasoning process can be effectively reduced.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to an information processing method, apparatus, device, and computer-readable storage medium. BACKGROUND

[0002] With the development of computers, various forms of electronic devices can greatly enrich people's daily life. For example, people can use electronic devices to interact in various ways, such as generating response content based on input content, and so on. SUMMARY

[0003] In a first aspect of the present disclosure, an information processing method is provided. The method comprises: obtaining input content of a model; determining, by using an attention unit of the model, first query features and first key features based on the input content; processing, by using a dimension reduction unit of the model, the first query features and the first key features to determine second query features and second key features; determining attention information based on the second query features and the second key features; and generating response content of the model for the input content based on the attention information.

[0004] In a second aspect of the present disclosure, an apparatus for information processing is provided. The apparatus comprises: an obtaining module configured to obtain input content of a model; a first determining module configured to determine, by using an attention unit of the model, first query features and first key features based on the input content; a processing module configured to process, by using a dimension reduction unit of the model, the first query features and the first key features to determine second query features and second key features; a second determining module configured to determine attention information based on the second query features and the second key features; and a generating module configured to generate response content of the model for the input content based on the attention information.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon a computer program, which is executable by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product comprises computer executable instructions that, when executed by a processor, implement the method according to the first aspect of the present disclosure.

[0008] It is to be understood that the particulars shown herein are by way of example and for purposes of illustrative discussion of the embodiments of the present disclosure only and are not intended to limit the scope of the present disclosure to the particular embodiments illustrated. Other BRIEF DESCRIPTION OF DRAWINGS

[0009] The above-mentioned and other features and advantages of various embodiments of this disclosure will become more apparent by reference to the following detailed description taken in conjunction with the accompanying drawings. In the drawings, like reference numerals designate like elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flow diagram showing an information processing process according to some embodiments of the present disclosure is shown;

[0012] Figure 3 An example flow diagram showing model training according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic block diagram showing an apparatus for information processing according to certain embodiments of the present disclosure is shown;

[0014] Figure 5 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It is to be understood that the drawings and embodiments of the present disclosure are merely for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0016] It is noted that the headings provided herein are not limitations of the various embodiments described herein. The various embodiments described throughout this document can be included under any heading. Additionally, embodiments described in any heading can be combined with any other embodiment described in the same heading and / or a different heading in any manner.

[0017] In the description of the embodiments of the present disclosure, the term "comprising" and its similar terms are understood to be open-ended, i.e., "including but not limited to". The term "based on" is understood to be "based, at least in part, on". The term "one embodiment" or "the embodiment" is understood to be "at least one embodiment". The term "some embodiments" is understood to be "at least some embodiments". Other explicit or implicit definitions can also be included below. The terms "first", "second", etc. can refer to different or same objects. Other explicit or implicit definitions can also be included below.

[0018] In the embodiments of the present disclosure, data of users, acquisition and / or use of data, etc. can be involved. These aspects all comply with corresponding laws and regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are performed on the premise that the user is aware of and confirms. Accordingly, when implementing the embodiments of the present disclosure, the type of data or information that can be involved, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations. The specific informing and / or authorization manner can vary according to actual situations and application scenarios, and the scope of the present disclosure is not limited in this aspect.

[0019] In the present specification and embodiments, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. The user refuses to process personal information other than the necessary information required for the basic function, which does not affect the user's use of the basic function.

[0020] Traditionally, the computational overhead of a language model in the inference process and the memory overhead of the key-value cache are generally large. Especially for long context inference scenarios, the computational overhead and memory overhead increase with the increase of the context length, which significantly limits the performance of memory-constrained systems, which affects the application of language models in low-resource (including edge devices) scenarios.

[0021] Embodiments of the present disclosure provide an information processing scheme. According to the scheme, input content of a model can be acquired. Further, a first query feature and a first key feature can be determined based on the input content by using an attention unit of the model. Further, a second query feature and a second key feature can be determined by processing the first query feature and the first key feature by using a dimension reduction unit of the model. Further, attention information can be determined based on the second query feature and the second key feature. Further, response content of the model for the input content can be generated based on the attention information.

[0022] Based on such a manner, the embodiments of the present disclosure can accurately estimate attention by reducing the dimensions of the query feature and the key feature, which not only ensures the accuracy of information processing, but also effectively reduces the computational and memory overhead of the model in the inference process.

[0023] Example Environment

[0024] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. As shown, the example environment 100 can include an electronic device 110. Figure 1

[0025] In some embodiments, the electronic device 110 can obtain input content of a model; determine, by using an attention unit of the model, a first query feature and a first key feature based on the input content; process the first query feature and the first key feature by using a dimension reduction unit of the model to determine a second query feature and a second key feature; determine attention information based on the second query feature and the second key feature; and generate response content of the model for the input content based on the attention information. The model 120 for processing the input content to generate the response content can be deployed on the electronic device 110, and can also be deployed on other devices, which will not be described here.

[0026] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media player, a multimedia tablet, a palmtop computer, a portable gaming terminal, a VR / AR device, a Personal Communication System (PCS) device, a personal navigation device, a Personal Digital Assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface to the user (such as "wearable" circuitry, etc.).

[0027] ​The electronic device 110 can also be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, a cloud server providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network, and basic cloud computing services such as big data and artificial intelligence platform, etc. The electronic device 110 may, for example, include a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc.

[0028] It should be understood that the structure and function of various embedded elements in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0029] Some example embodiments of the present disclosure will be described below with continuous reference to the drawings.

[0030] Example Process

[0031] Figure 2 A flowchart of an information processing process 200 according to some embodiments of the present disclosure is shown. The process 200 can be implemented at the electronic device 110. The process 200 will be described below with reference to Figure 1 The process 200.

[0032] At block 210, the electronic device 110 obtains input content for a model.

[0033] In some embodiments, the input content can be any suitable type of content, such as text, image, audio, video, etc. This input content is the content to be input to the model and processed by the model.

[0034] In some embodiments, the model can be any suitable machine learning model. As an example, the model can include an autoregressive model or other suitable generative model. Such models can be applied to scenarios such as text generation, image generation, video generation, audio generation, and / or multi-modal content generation.

[0035] In some examples, the input content provided to the model can also be referred to as a prompt. As an example, in the scenario of generating an image, the prompt can be used to describe the image to be generated, e.g., “a cat playing a guitar”.

[0036] At block 220, the electronic device 110 determines, based on the input content, first query features and first key features using an attention unit of the model.

[0037] In some embodiments, the attention unit can be a component in the model for implementing a self-attention mechanism. The self-attention mechanism is used to indicate that the model dynamically focuses on information in different parts of the input content when processing the input content, thereby capturing the dependencies between the parts.

[0038] In some embodiments, the electronic device 110 can convert the input content into a set of tokens, each token representing a part of the input content, such as a word or a character, and the like.

[0039] Further, the electronic device 110 can convert the set of tokens into a set of vectors. Further, the electronic device 110 can determine, through a predetermined linear transformation, a query feature (Query), a key feature (Key), and a value feature (Value) corresponding to each token.

[0040] In some embodiments, the first query feature can be a query corresponding to a token, and the first key feature includes a set of key vectors, which are keys corresponding to the set of tokens.

[0041] At block 230, the electronic device 110 processes the first query feature and the first key feature using the dimension reduction unit of the model to determine a second query feature and a second key feature.

[0042] In some embodiments, the dimension reduction unit is used to perform dimension reduction processing on the features to convert the original features corresponding to the feature dimensions into new features with lower feature dimensions.

[0043] To improve the efficiency of dimension reduction, in some embodiments, the dimension reduction unit can include a first unit for performing dimension reduction processing on the first query feature and a second unit for performing dimension reduction processing on the first key feature. The two units can perform parallel dimension reduction processing on different features, which can effectively improve the efficiency of dimension reduction. In some embodiments, the first unit and the second unit are any appropriate dimension reduction units, such as the first unit can be composed of a first linear layer and the second unit can be composed of a second linear layer. The first linear layer and the second linear layer can support the conversion of high-dimensional features into low-dimensional features.

[0044] In some embodiments, the electronic device 110 can provide the first query feature to the first unit to determine the second query feature. In some embodiments, the first feature dimension of the first query feature is greater than the second feature dimension of the second query feature.

[0045] In some embodiments, the electronic device 110 can provide the first key feature to the second unit to determine the second key feature. In some embodiments, the third feature dimension of the first key feature is greater than the fourth feature dimension of the second key feature.

[0046] The following describes the training process of the dimension reduction unit, taking the training of the dimension reduction unit by the server as an example.

[0047] In some embodiments, the server can obtain sample query features and sample key features determined based on sample content. In some embodiments, the sample content can be any appropriate type of content, such as text, images, audio, video, and the like.

[0048] Further, the server can process the sample query features and the sample key features by using the dimension reduction unit to generate compressed query features and compressed key features. In some embodiments, the server can provide the sample query features to a first unit of the dimension reduction unit to determine the compressed query features. The feature dimension of the sample query features is greater than the second feature dimension of the compressed query features.

[0049] In some embodiments, the server can provide the sample key features to a second unit of the dimension reduction unit to determine the compressed key features. In some embodiments, the feature dimension of the sample key features is greater than the feature dimension of the compressed key features.

[0050] Further, the server can determine first attention information based on the sample query features and the sample key features.

[0051] For example, the server can determine the first attention information based on the following formula:

[0052]

[0053] wherein A1 is the first attention information, q is the sample query features, k is the sample key features, and D is the dimension of the high-dimensional space, i.e., the dimension of the space before dimension reduction.

[0054] Further, the server can determine second attention information based on the compressed query features and the compressed key features.

[0055] For example, the server can determine the second attention information based on the following formula:

[0056]

[0057] wherein A1 is the second attention information, is the compressed query features, is the compressed key features, and d k is the dimension of the low-dimensional space, i.e., the dimension of the space after dimension reduction.

[0058] Further, the server can train the dimension reduction unit based on the difference between the first attention information and the second attention information.

[0059] Specifically, the server can determine the loss value based on the first attention information, the second attention information, and a predetermined loss function. The loss function can be any suitable loss function, such as a cross-entropy function. Further, the server can train the dimension reduction unit based on the loss value until a predetermined training completion condition is reached. The predetermined training completion condition can be any suitable condition, such as the loss value reaching a minimum value, a training duration reaching a threshold value, a training number reaching a threshold value, and the like, which will not be repeated here.

[0060] For example, the server can determine the loss value based on the following formula:

[0061]

[0062] where loss is the loss value, CE denotes cross-entropy, qK T ∈ R 1×L denotes the first attention information, and denotes the second attention information determined based on the dimension-reduced and qK T , which is normalized to a probability distribution.

[0063] Figure 3 An example flowchart of model training according to some embodiments of the present disclosure is shown, which will now be described with respect to Figure 3 .

[0064] As shown in Figure 3 , the model can obtain a sample content 305. Further, an attention unit 310 can determine a sample query feature 325 and a sample key feature 330 based on the sample content 305. Additionally, the attention unit 310 can also determine a first attention information 315 based on the sample query feature 325 and the sample key feature 330.

[0065] Further, the sample query feature 325 and the sample key feature 330 can be provided to a dimension reduction unit 320 to determine a second attention information 335.

[0066] Specifically, as shown in Figure 3 , the dimension reduction unit 320 can include a double tower model, which can include a first unit 340 and a second unit 345. The first unit 340 can perform dimension reduction processing on the sample query feature to obtain a compressed query feature. Correspondingly, the second unit 345 can perform dimension reduction processing on the sample key feature to obtain a compressed key feature.

[0067] In some embodiments, the second attention information 335 can be determined based on the compressed key features and the compressed query features. Further, the dimension reduction model 320 can be trained based on the difference between the first attention information and the second attention information. For example, the parameters of the first unit 340 and the second unit 345 can be adjusted based on

[0068] In some embodiments, multiple attention units are included in the model, and the redundant dimensions are different between the respective attention units, where the redundant dimensions represent that not all generated features or dimensions contribute substantially to the final output in the attention mechanism of the model, i.e., some features that do not provide additional information can have little impact on the decision of the model. In addition, there are differences in the degree of support for reducing dimensions by different attention units, such as some attention units can not be reduced to 32 dimensions, while other attention units can be reduced to 32 dimensions without introducing substantial loss. However, it is challenging to adjust the training parameters and settings of each attention unit individually, which significantly increases the complexity of model training.

[0069] To avoid the problem that training adaptive ranks for each attention unit will significantly increase the complexity of model training and reduce practicability, in some embodiments, the present disclosure can obtain multiple candidate dimension reduction units in advance, so that the electronic device 110 can select a better attention unit from the multiple candidate dimension reduction units based on different attention units.

[0070] The process of determining the dimension reduction unit corresponding to the attention unit by the server is described below.

[0071] In some embodiments, the server can obtain multiple candidate dimension reduction units for the attention unit. In some embodiments, the multiple candidate dimension reduction units are associated with different attributes. In some embodiments, the attributes can include but are not limited to model structure, learning parameter, dimension reduction parameter.

[0072] In some embodiments, the model structure represents the structure of the neural network adopted by the candidate dimension reduction unit, and this model result can indicate the number of layers included in the candidate dimension reduction unit, the number of neurons in each layer, etc.

[0073] In some embodiments, the learning parameter can represent any appropriate parameter that can determine the behavior of the candidate dimension reduction unit learned during the training process, such as but not limited to weight, bias, etc.

[0074] In some embodiments, the dimension reduction parameter can be any appropriate parameter that can affect the feature dimension reduction effect.

[0075] ​Further, the server can generate evaluation information for the plurality of candidate dimension reduction units. The evaluation information can be any suitable evaluation data or index for evaluating the candidate dimension reduction units, such as but not limited to the dimension reduction capability, the dimension reduction accuracy, the efficiency, the resource consumption, and the like.

[0076] As an example, the server can determine attention difference information associated with the candidate dimension reduction unit, the attention difference information indicating a difference between third attention information and fourth attention information, the third attention information being determined by applying the candidate dimension reduction unit, the fourth attention information being independent of the candidate dimension reduction unit. In some embodiments, the server can construct a test sample, the test sample can indicate first test query features and first test key features. Further, the server can provide the test sample to the candidate dimension reduction unit to determine second test query features and second test key features. Further, the server can determine the fourth attention information based on the first test query features and the first test key features, and determine the third attention information based on the second test query features and the second test key features. Further, the server can determine the attention difference information between the third attention information and the fourth attention information.

[0077] Further, the server can determine evaluation information of the candidate dimension reduction unit based on the attention difference information. Since the present disclosure expects that the attention information determined after the dimension reduction processing of the key features and the query features is close to the attention information determined before the dimension reduction processing, which can ensure the accuracy of the information processing, the smaller the attention difference information is, the higher the evaluation of the candidate dimension reduction unit is.

[0078] For example, the server can determine the evaluation information of the candidate dimension reduction unit based on the following formula:

[0079]

[0080] wherein A m is the attention difference, softmax(qK T V is the fourth attention information, is the third attention information; is the normalized attention difference corresponding to the candidate dimension reduction unit m, M i is the plurality of candidate dimension reduction units.

[0081] Further, the server can determine the dimension reduction unit for the attention unit from the plurality of candidate dimension reduction units based on the evaluation information. As an example, the server can determine the candidate dimension reduction unit whose evaluation information satisfies a preset evaluation condition as the dimension reduction unit for the attention unit. As an example, the server can construct a test set, which can include a plurality of test samples. For each of the plurality of candidate dimension reduction units, the server can determine a plurality of attention difference information based on the plurality of test samples and the candidate dimension reduction unit. Further, the server can determine, for the candidate dimension reduction unit, a number of test samples in the plurality of test samples whose attention difference information is less than a predetermined difference threshold. Further, the server can count the candidate dimension reduction units corresponding to the largest number among the plurality of candidate dimension reduction units, and determine the candidate dimension reduction unit as the dimension reduction unit for the attention unit.

[0082] As an example, the server can determine a group of candidate dimension reduction units whose evaluation information satisfies a preset condition from the plurality of candidate dimension reduction units. The preset condition can be any appropriate condition, such as the preset condition indicating that the group of candidate dimension reduction units is a preset number of candidate dimension reduction units with the top evaluation information.

[0083] As an example, the server can construct a test set, which can include a plurality of test samples. For each of the plurality of candidate dimension reduction units, the server can determine a plurality of attention difference information based on the plurality of test samples and the candidate dimension reduction unit. Further, the server can determine, for the candidate dimension reduction unit, a number of test samples in the plurality of test samples whose attention difference information is less than a predetermined difference threshold. Further, the server can sort the plurality of candidate dimension reduction units in descending order of the number based on the number corresponding to the plurality of candidate dimension reduction units. Further, the server can determine a preset number of candidate dimension reduction units with the top ranking as the group of candidate dimension reduction units.

[0084] For example, the server can determine the group of candidate dimension reduction units based on the following formula:

[0085]

[0086] wherein M i is a unit set composed of the plurality of candidate dimension reduction units, S t is the group of candidate dimension reduction units, m is a candidate dimension reduction unit m in the group of candidate dimension reduction units, is the normalized attention difference corresponding to the candidate dimension reduction unit m, and t is a preset threshold.

[0087] Further, the server can determine the dimension reduction unit for the attention unit based on the dimension compression ratios corresponding to the set of candidate dimension reduction units. As an example, the server can determine the candidate dimension reduction unit with the highest dimension compression ratio in the set of candidate dimension reduction units as the dimension reduction unit for the attention unit. As an example, the server can determine the candidate dimension reduction unit satisfying a predetermined compression ratio condition in the set of candidate dimension reduction units as the dimension reduction unit for the attention unit.

[0088] For example, the server can determine the dimension reduction unit for the attention unit based on the following formula:

[0089]

[0090] wherein S t is the set of candidate dimension reduction units, is a candidate dimension reduction unit in the set of candidate dimension reduction units, is the minimum dimension corresponding to the dimension reduction, d(m) is the dimension reduced by the candidate dimension reduction unit m, S d is the dimension reduction unit for the attention unit.

[0091] In some embodiments, the server can determine the dimension reduction unit for the attention unit by comparing the first evaluation information of the first candidate dimension reduction unit and the second evaluation information of the second candidate dimension reduction unit in response to the first candidate dimension reduction unit and the second candidate dimension reduction unit corresponding to the same dimension compression ratio. As an example, the server can determine the candidate dimension reduction unit with higher evaluation information in the first candidate dimension reduction unit and the second candidate dimension reduction unit based on the first evaluation information and the second evaluation information. Further, the server can determine the candidate dimension reduction unit with higher evaluation information as the dimension reduction unit for the attention unit.

[0092] For example, the server can determine the dimension reduction unit for the attention unit based on the following formula:

[0093]

[0094] wherein, is the candidate dimension reduction unit is the normalized attention difference corresponding to the candidate dimension reduction unit, is a candidate dimension reduction unit in the set of candidate dimension reduction units, d is the dimension reduction unit for the attention unit, * is the dimension reduction unit for the attention unit.

[0095] In block 240, the electronic device 110 determines the attention information based on the second query feature and the second key feature.

[0096] In some embodiments, the second key feature includes a set of key vectors, where different key vectors correspond to different tokens.

[0097] As an example, the electronic device 110 can determine the attention information based on the set of key vectors and the second query feature.

[0098] For example, the electronic device 110 can determine the attention information based on the following formula:

[0099]

[0100] where A is the attention information, Q is the second query feature, K is the second key feature, D k is the dimension of the reduced dimension space.

[0101] In some embodiments, the electronic device 110 can determine a predetermined number of key vectors from the second key feature based on the second query feature. As an example, the electronic device 110 can determine the relevance of the second query feature to the set of key vectors based on the second query feature. Further, the electronic device 110 can sort the set of key vectors in order of relevance from high to low based on the relevance of the second query feature to the set of key vectors. Further, the electronic device 110 can determine the top pre-set number of key vectors as the plurality of key vectors. For example, the electronic device 110 can determine the top 2 key vectors as the plurality of key vectors. Figure 3 As an example, at 350, the top-K key vectors can be determined from the second query feature output by the first unit 340.

[0102] Further, the electronic device 110 can determine the attention information based on the plurality of key vectors. Specifically, the electronic device 110 can determine the attention information based on the plurality of key vectors and the second query feature. For example, the electronic device 110 can determine the attention information based on the following formula: Figure 3 As an example, the model can determine the attention information 360 based on the selected K key vectors.

[0103] At block 250, the electronic device 110 generates response content for the input content based on the attention information.

[0104] In some embodiments, the electronic device 110 can determine the importance of the value vector of each token in generating the response content for the input content based on the attention information, which can be indicated based on a probability distribution determined based on the attention information.

[0105] Further, the electronic device 110 can weight and sum the value vectors of each token based on the probability distribution to obtain an aggregated vector.

[0106] Furthermore, the electronic device 110 may generate response content of the model to the input content based on the aggregated vector. The response content may be any appropriate type of content, such as text, voice, image, etc.

[0107] Based on this approach, the embodiments of the present disclosure can accurately estimate attention by reducing the dimensions of query features and key features, which not only ensures the accuracy of information processing, but also effectively reduces the computational overhead and memory overhead of the model during the inference process.

[0108] Example Apparatus and Device

[0109] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 : A schematic structural block diagram of an apparatus 400 for information processing according to certain embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in the electronic device 110 discussed above. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0110] like Figure 4 As shown, the device 400 includes an acquisition module 410, which is configured to acquire the input content of the model; a first determination module 420, which is configured to use the attention unit of the model to determine the first query feature and the first key feature based on the input content; a processing module 430, which is configured to use the dimensionality reduction unit of the model to process the first query feature and the first key feature to determine the second query feature and the second key feature; a second determination module 440, which is configured to determine the attention information based on the second query feature and the second key feature; and a generation module 450, which is configured to generate the response content of the model to the input content based on the attention information.

[0111] In some embodiments, the dimensionality reduction unit includes a first unit and a second unit, and the processing module 430 is further configured to: provide the first query feature to the first unit to determine the second query feature; and provide the first key feature to the second unit to determine the second key feature.

[0112] In some embodiments, the dimensionality reduction unit is trained based on the following process: obtaining sample query features and sample key features determined based on sample content; using the dimensionality reduction unit to process the sample query features and sample key features to generate compressed query features and compressed key features; determining first attention information based on the sample query features and sample key features; determining second attention information based on the compressed query features and compressed key features; and training the dimensionality reduction unit based on the difference between the first attention information and the second attention information.

[0113] In some embodiments, the dimension reduction unit is determined based on a process comprising: obtaining a plurality of candidate dimension reduction units for the attention unit; generating evaluation information for the plurality of candidate dimension reduction units; and determining the dimension reduction unit for the attention unit from the plurality of candidate dimension reduction units based on the evaluation information.

[0114] In some embodiments, the plurality of candidate dimension reduction units are associated with different attributes, the attributes indicating at least one of: a model structure; a learning parameter; a dimension reduction parameter.

[0115] In some embodiments, generating the evaluation information for the plurality of candidate dimension reduction units comprises: determining attention difference information associated with a candidate dimension reduction unit, the attention difference information indicating a difference between third attention information determined by applying the candidate dimension reduction unit and fourth attention information independent of the candidate dimension reduction unit; and determining the evaluation information for the candidate dimension reduction unit based on the attention difference information.

[0116] In some embodiments, determining the dimension reduction unit for the attention unit from the plurality of candidate dimension reduction units based on the evaluation information comprises: determining a set of candidate dimension reduction units from the plurality of candidate dimension reduction units for which the evaluation information satisfies a preset condition; and determining the dimension reduction unit for the attention unit based on dimension compression ratios corresponding to the set of candidate dimension reduction units.

[0117] In some embodiments, determining the dimension reduction unit for the attention unit based on the dimension compression ratios corresponding to the set of candidate dimension reduction units comprises: in response to a first candidate dimension reduction unit and a second candidate dimension reduction unit corresponding to a same dimension compression ratio, determining the dimension reduction unit for the attention unit by comparing first evaluation information of the first candidate dimension reduction unit and second evaluation information of the second candidate dimension reduction unit.

[0118] In some embodiments, the second determining module 440 is further configured to: determine a predetermined number of key vectors from the second key feature based on the second query feature; and determine the attention information based on the plurality of key vectors.

[0119] In some embodiments, a first feature dimension of the first query feature is greater than a second feature dimension of the second query feature; and a third feature dimension of the first key feature is greater than a fourth feature dimension of the second key feature.

[0120] The units included in the apparatus 400 can be implemented utilizing various means, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, e.g., machine-executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units in the apparatus 400 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0121] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the disclosure can be implemented is shown. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1 The electronic device 110 shown.

[0122] As Figure 5 The electronic device 500 is in the form of a general-purpose electronic device, as shown. Components of the electronic device 500 can include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing units 510 can be actual or virtual processors and capable of executing various processing in accordance with programs stored in the memory 520. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel to improve the processing power of the electronic device 500.

[0123] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any available media that is accessible by the electronic device 500 and includes both volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable media and can include machine-readable media, such as a flash drive, a magnetic disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and that can be accessed by the electronic device 500.

[0124] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0125] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0126] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 500 may also communicate with one or more external devices (not shown) via communication unit 540 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with electronic device 500, or with any device that allows electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0127] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0128] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0129] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0130] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0131] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0132] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for information processing, comprising: Get the input content of the model; Determining, using an attention unit of the model, a first query feature and a first key feature based on the input content; Processing the first query feature and the first key feature using a dimensionality reduction unit of the model to determine a second query feature and a second key feature; determining attention information based on the second query feature and the second key feature; as well as Based on the attention information, generate response content of the model to the input content.

2. The method according to claim 1, wherein the dimensionality reduction unit includes a first unit and a second unit, and processing the first query feature and the first key feature using the dimensionality reduction unit of the model to determine the second query feature and the second key feature comprises: providing the first query feature to the first unit to determine the second query feature; as well as The first key characteristic is provided to the second unit to determine the second key characteristic.

3. The method according to claim 1, wherein the dimensionality reduction unit is trained based on the following process: Obtaining sample query features and sample key features determined based on sample content; Processing the sample query feature and the sample key feature using the dimensionality reduction unit to generate a compressed query feature and a compressed key feature; determining first attention information based on the sample query feature and the sample key feature; determining second attention information based on the compressed query feature and the compressed key feature; as well as The dimensionality reduction unit is trained based on the difference between the first attention information and the second attention information.

4. The method according to claim 1, wherein the dimensionality reduction unit is determined based on the following process: Obtaining multiple candidate dimensionality reduction units for the attention unit; generating evaluation information for the plurality of candidate dimensionality reduction units; and Based on the evaluation information, the dimensionality reduction unit for the attention unit is determined from the multiple candidate dimensionality reduction units.

5. The method according to claim 4, wherein the plurality of candidate dimensionality reduction units are associated with different attributes, wherein the attributes indicate at least one of the following: Model structure; Learning parameters; Dimensionality reduction parameters.

6. The method according to claim 4, wherein generating evaluation information for the plurality of candidate dimensionality reduction units comprises: determining attention difference information associated with a candidate dimensionality reduction unit, the attention difference information indicating a difference between third attention information and fourth attention information, the third attention information being determined by applying the candidate dimensionality reduction unit, and the fourth attention information being independent of the candidate dimensionality reduction unit; as well as Based on the attention difference information, the evaluation information of the candidate dimensionality reduction unit is determined.

7. The method according to claim 4, wherein determining the dimensionality reduction unit for the attention unit from the plurality of candidate dimensionality reduction units based on the evaluation information comprises: Determining a group of candidate dimensionality reduction units whose evaluation information meets a preset condition from the multiple candidate dimensionality reduction units; as well as Based on the dimensionality compression ratio corresponding to the set of candidate dimensionality reduction units, the dimensionality reduction unit used for the attention unit is determined.

8. The method according to claim 7, wherein determining the dimensionality reduction unit for the attention unit based on the dimensionality compression ratios corresponding to the set of candidate dimensionality reduction units comprises: In response to the first candidate dimensionality reduction unit and the second candidate dimensionality reduction unit corresponding to the same dimensionality compression ratio, the dimensionality reduction unit for the attention unit is determined by comparing the first evaluation information of the first candidate dimensionality reduction unit and the second evaluation information of the second candidate dimensionality reduction unit.

9. The method of claim 1 , wherein determining attention information based on the second query feature and the second key feature comprises: determining a predetermined number of multiple key vectors from the second key feature based on the second query feature; as well as Based on the multiple key vectors, the attention information is determined.

10. The method of claim 1, wherein: A first feature dimension of the first query feature is greater than a second feature dimension of the second query feature; and The third feature dimension of the first key feature is greater than the fourth feature dimension of the second key feature.

11. An apparatus for information processing, comprising: An acquisition module, configured to obtain input content of the model; a first determination module, configured to determine a first query feature and a first key feature based on the input content using an attention unit of the model; a processing module configured to process the first query feature and the first key feature using a dimensionality reduction unit of the model to determine a second query feature and a second key feature; a second determining module configured to determine attention information based on the second query feature and the second key feature; as well as A generation module is configured to generate response content of the model to the input content based on the attention information.

12. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images