Information processing method, apparatus, computer-readable storage medium, and computer program product

Abstract

The present disclosure relates to an information processing method and device, a computer readable storage medium and a computer program product, and relates to the technical field of computer. The method comprises: using a machine learning model to calculate a query vector and a key vector of each of a plurality of labeled data included in to-be-processed information; calculating a first attention weight of each of the plurality of labeled data according to part of the features in the query vector and part of the features in the key vector; and determining a processing result of the to-be-processed information according to the first attention weight. The technical solution of the present disclosure can adapt to different to-be-processed information, thereby improving the accuracy of computer processing information.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an information processing method, an information processing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] During the inference and decoding process of machine learning models, especially in scenarios with long texts, the main factor limiting inference speed is the self-attention module. To address this technical issue, token (labeled data) sparsification methods have been proposed to improve the inference efficiency of machine learning models.

[0003] In related technologies, several fixed tokens can be retained during the reasoning process to improve the reasoning efficiency of the machine learning model. Summary of the Invention

[0004] According to some embodiments of the present disclosure, an information processing method is provided, including: using a machine learning model to calculate a query vector and a key vector for each of a plurality of labeled data included in the information to be processed; calculating a first attention weight for each of the labeled data based on partial features in the query vector and partial features in the key vector; and determining a processing result of the information to be processed based on the first attention weight.

[0005] In some embodiments, determining the processing result of the information to be processed based on the first attention weight includes: determining key tag data from multiple tag data based on the first attention weight; and determining the processing result using a machine learning model based on the key tag data.

[0006] In some embodiments, determining the key tag data among the plurality of tag data according to the first attention weight includes: determining at least one tag data having the largest first attention weight as the key tag data.

[0007] In some embodiments, determining the processing result of the information to be processed based on the first attention weight includes: calculating the second attention weight of each tagged data based on the query vector of each tagged data and the key vector of the key tagged data; and determining the processing result of the information to be processed based on the second attention weight using a machine learning model.

[0008] In some embodiments, multiple processing units in a machine learning model process information to be processed based on the same key tag data, and the multiple processing units are arranged adjacent to each other in the machine learning model.

[0009] In some embodiments, the multiple processing units are multiple attention heads arranged adjacent to each other in a machine learning model, and / or multiple layers arranged adjacent to each other in a machine learning model.

[0010] In some embodiments, the processing unit includes a first processing unit and a second processing unit, and determining the processing result using a machine learning model based on key tag data includes: using the second processing unit to determine the processing result based on the key tag data determined in the first processing unit.

[0011] In some embodiments, a predetermined number of adjacent query vectors correspond to the same key tag data.

[0012] In some embodiments, a preset number of adjacent query vectors include a first query vector and a second query vector, and determining the key tag data in multiple tag data based on the first attention weight includes: determining the key tag data corresponding to the first query vector based on the first attention weight corresponding to the first query vector; and determining the key tag data corresponding to the first query vector as the key tag data corresponding to the second query vector.

[0013] According to other embodiments of the present disclosure, an information processing device is provided, including: a vector calculation module for calculating, by using a machine learning model, a query vector and a key vector for each of a plurality of labeled data included in the information to be processed; an attention calculation module for calculating a first attention weight for each of the labeled data based on partial features in the query vector and partial features in the key vector; and a processing module for determining a processing result of the information to be processed based on the first attention weight.

[0014] In some embodiments, the processing module determines key tag data among the multiple tag data based on the first attention weight; and determines the processing result using the machine learning model based on the key tag data.

[0015] In some embodiments, the processing module determines at least one labeled data with the largest first attention weight as key labeled data.

[0016] In some embodiments, the attention calculation module calculates the second attention weight of each tagged data based on the query vector of each tagged data and the key vector of the key tagged data; the processing module determines the processing result of the information to be processed using the machine learning model based on the second attention weight.

[0017] In some embodiments, multiple processing units in a machine learning model process information to be processed based on the same key tag data, and the multiple processing units are arranged adjacent to each other in the machine learning model.

[0018] In some embodiments, the multiple processing units are multiple attention heads arranged adjacent to each other in a machine learning model, and / or multiple layers arranged adjacent to each other in a machine learning model.

[0019] In some embodiments, the processing unit includes a first processing unit and a second processing unit, and the processing module uses the second processing unit to determine the processing result according to the key tag data determined in the first processing unit.

[0020] In some embodiments, a predetermined number of adjacent query vectors correspond to the same key tag data.

[0021] In some embodiments, a preset number of adjacent query vectors include a first query vector and a second query vector. The processing module determines the key tag data corresponding to the first query vector based on the first attention weight corresponding to the first query vector, and determines the key tag data corresponding to the first query vector as the key tag data corresponding to the second query vector.

[0022] According to some embodiments of the present disclosure, an information processing device is provided, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the information processing method of any embodiment of the present disclosure based on instructions stored in the memory.

[0023] According to some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the information processing method of any embodiment of the present disclosure is performed.

[0024] According to some embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, cause the processor to perform the information processing method according to any one of the embodiments of the present disclosure.

[0025] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The following describes embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation to the present disclosure. In the accompanying drawings:

[0027] Figure 1 Flowcharts showing some embodiments of the information processing method of the present disclosure;

[0028] Figures 2a to 2c A schematic diagram illustrating some embodiments of key tag data of the present disclosure;

[0029] Figure 3 A block diagram illustrating some embodiments of the information processing device of the present disclosure;

[0030] Figure 4 A block diagram showing some other embodiments of the information processing device of the present disclosure;

[0031] Figure 5 A block diagram showing still further embodiments of the information processing device of the present disclosure.

[0032] It should be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not necessarily drawn to scale. The same or similar reference numerals are used throughout the drawings to indicate the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. DETAILED DESCRIPTION

[0033] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. It should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein.

[0034] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in different orders and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0035] The term “including” and its variations used in the present disclosure are open terms that include at least the following elements / features but do not exclude other elements / features, that is, “including but not limited to.” The term “based on” means “at least in part based on.”

[0036] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules, or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules, or units. Unless otherwise specified, concepts such as "first" and "second" are not intended to imply that the objects described in such a manner must be in a given order in time, space, ranking, or any other manner.

[0037] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0039] The following detailed description of the embodiments of the present disclosure is provided in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. In addition, in one or more embodiments, specific features, structures, or characteristics may be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0040] The inventors of the present disclosure have discovered that the above-mentioned related technologies have the following problems: they cannot adapt to different information to be processed, resulting in poor accuracy in computer processing of information.

[0041] In view of this, the present disclosure proposes an information processing technology solution that can adapt to different information to be processed, thereby improving the accuracy of computer processing information.

[0042] As mentioned earlier, although sparsification methods that retain several important tokens (such as Stream and H2O) can reduce the amount of computation to a certain extent, the removed tokens are fixed, resulting in a decrease in the computer's reasoning accuracy.

[0043] To address the above technical issues, the disclosed technical solution uses only some features from the query and key vectors of the information to be processed to calculate a simplified attention weight (atten_weight) as the basis for subsequent reasoning. This not only effectively reduces the computing and storage resources required for computer reasoning, improving the computer's reasoning efficiency, but also can adapt to different information to be processed, thereby improving the accuracy of the computer's information processing.

[0044] For example, the following embodiments may be used to implement the technical solutions of the present disclosure.

[0045] Figure 1 Flowcharts showing some embodiments of the information processing method of the present disclosure.

[0046] like Figure 1 As shown, in step 110, a machine learning model is used to calculate a query vector (query) and a key vector (key) for each of the multiple tokens (tokens) included in the information to be processed. For example, the information to be processed may be natural language information (such as text information, voice information, etc.), and a token may be a word, a character, or a part of a word in the natural language information.

[0047] In some embodiments, the machine learning model may include a self-attention module; the self-attention module may be used to calculate the query vector and key vector of each labeled data according to the feature vector of each labeled data. For example, the dimensions of query and key are query∈RB,1,N , key∈R B,S,N , B is the batch size (the size of a batch), S is the sequence length of the information to be processed, and N is the feature dimension.

[0048] In step 120, a first attention weight is calculated for each labeled data item based on some features in the query vector and some features in the key vector. For example, a preset number or a preset percentage of features can be sampled from query to generate query', and a preset number (e.g., N / 8, etc.) or a preset percentage (e.g., 1 / 8, etc.) of features can be sampled from key to generate key'; and the first attention weight atten_weights is calculated based on query' and key'.

[0049] For example, extract N / 8-dimensional features from the N-dimensional features of query to generate query', extract N / 8-dimensional features from the N-dimensional features of key to generate key', and then calculate the first attention weight atten_weights according to the following formula:

[0050] atten_weights = query '×key' T

[0051] In this way, by using some features from the query and key vectors, we obtain compressed attention weights for subsequent computer information processing, achieving dimensionality compression and reducing computational complexity. This not only effectively reduces the computing and storage resources required for computer reasoning, improving the efficiency of computer reasoning, but also allows for adaptability to different types of information to be processed, thereby improving the accuracy of computer processing.

[0052] In step 130, a processing result of the information to be processed is determined based on the first attention weight. For example, the first attention weight can be used to perform subsequent processing in the machine learning model encoder and decoder, thereby improving the computer's reasoning efficiency.

[0053] In some embodiments, key tagged data from multiple tagged data is determined based on the first attention weights; based on the key tagged data (key tokens), a processing result is determined using a machine learning model. In this way, the first attention weights reduce computational complexity while retaining sufficient information. Using the first attention weights, key tokens in the information to be processed can be quickly and accurately identified, thereby improving the efficiency and accuracy of the computer's reasoning.

[0054] In some embodiments, at least one tag data with the largest first attention weight is determined as key tag data. For example, the information to be processed includes a total of L tokens, and the atten_weights of each token is calculated in step 120 to obtain L atten_weights; the L atten_weights can be sorted according to size, such as sorting in descending order to obtain the sorting result {atten_weights1, atten_weights2…atten_weights l …atten_weights L}; You can select the K largest atten_weights in the first K positions in the sorting result, that is, atten_weights1~atten_weights K , take these K largest atten_weights token1~token L The corresponding token serves as the key token.

[0055] In this way, only a small number of tokens with the highest attention are selected from a large number of tokens as key tokens for subsequent processing. These key tokens are the most representative and influential parts of the query within the context of the information being processed. Processing information based on key tokens reduces computational complexity while retaining sufficient information, allowing the computer to complete the reasoning process quickly and accurately, thereby improving reasoning efficiency and accuracy.

[0056] In some embodiments, the second attention weight of each tagged data is calculated based on the query vector of each tagged data and the key vector of the key tagged data; and based on the second attention weight, the processing result of the information to be processed is determined using a machine learning model.

[0057] For example, after obtaining key tokens online, you can use the query with complete dimensional features for each token in the information to be processed and the key' with partial dimensional features for each key token to calculate the second attention weight atten_weights'. The key' of non-key tokens is not involved in the calculation of atten_weights'. For example, you can set the key' of non-key tokens to 0 to simplify the calculation process.

[0058] In this way, only the key tokens that are most representative and influential to the query in the context of the information to be processed are selected to participate in the calculation of attention weights, which not only reduces the computational complexity but also retains sufficient information, allowing the computer to complete the reasoning process quickly and accurately, thereby improving the reasoning efficiency and accuracy.

[0059] In the above embodiment, a dynamic search method for key tokens in the information to be processed is proposed. In the self-attention unit of the machine learning model, a simplified attention weight is calculated using partial features in the query vector and key vector of each token; key tokens are selected based on this simplified attention weight for subsequent reasoning. Compared to static sparse token technology solutions, this improves the computer's information processing efficiency, reduces computational complexity, and thus increases the computer's information processing speed, while maintaining the same sparsity ratio.

[0060] For example, the information to be processed can be interactive information, such as text or voice, input by a user into a computer through an interactive interface such as an interactive platform. The processing result output by the machine learning model can be a response to the interactive information. The technical solutions in any of the above embodiments not only effectively reduce the computing and storage resources required for computer reasoning, but also improve the effectiveness and efficiency of human-computer interaction.

[0061] The following examples illustrate a technical solution for sharing key tokens, so as to avoid the need to re-evaluate the key token of each query for each calculation, thereby reducing the cost of computer resources and further improving the effect of computer reasoning.

[0062] For example, a machine learning model includes multiple processing units, such as multiple heads (attention heads), multiple layers, etc. Each layer can include multiple heads. Figures 2a to 2c This example illustrates the similarity of key tokens between different queries, different heads, and different layers.

[0063] Figures 2a to 2c Schematic diagram illustrating some embodiments of key tag data of the present disclosure.

[0064] like Figures 2a to 2c The following figure shows the attention weights calculated for different queries and keys in the first, sixteenth, and thirty-second layers of the machine learning model. The vertical axis of the left image represents the query sequence, and the horizontal axis represents the key sequence. Lower grayscale (i.e., brighter) values ​​in the left image correspond to higher attention weights, indicating more important tokens. The right image shows the relationship between attention weights and grayscale values. Each query is represented by a horizontal line, and the lowest grayscale values ​​on that line represent the key tokens corresponding to that query.

[0065] from Figures 2a to 2cAs can be seen, the key tokens corresponding to adjacent queries are consistent. Based on this, key tokens with high cosine similarity can be shared across queries, heads, and layers based on the cosine similarity between key tokens. This reduces the additional computational overhead associated with dynamic key token searches, improving processing efficiency while ensuring accurate computer information processing.

[0066] Based on the above analysis, key tokens can be shared through the following embodiments.

[0067] In some embodiments, multiple processing units in a machine learning model process information to be processed based on the same key labeled data, and the multiple processing units are arranged adjacent to each other in the machine learning model. For example, the multiple processing units are multiple attention heads arranged adjacent to each other in the machine learning model, and / or multiple layers arranged adjacent to each other in the machine learning model.

[0068] For example, the processing unit includes a first processing unit and a second processing unit, and the second processing unit is used to determine a processing result according to the key tag data determined in the first processing unit.

[0069] For example, a preset number (e.g., 8, etc.) of adjacent heads can be configured to share a set of key tokens using the method described in any of the above embodiments. A preset number (e.g., 8, etc.) of adjacent layers can also be configured to share a set of key tokens using the method described in any of the above embodiments.

[0070] In some embodiments, a preset number of adjacent query vectors correspond to the same key tag data. For example, the preset number of adjacent query vectors include a first query vector and a second query vector, and the key tag data corresponding to the first query vector is determined based on a first attention weight corresponding to the first query vector; and the key tag data corresponding to the first query vector is determined as the key tag data corresponding to the second query vector.

[0071] For example, a preset number (such as 2, etc.) of adjacent query attention heads can be configured to share a set of key tokens using the method in any of the above embodiments.

[0072] In the above embodiment, based on the similarity between key tokens of different queries, different heads, and different layers, key tokens are shared at multiple levels (e.g., query, head, and layer) of the machine learning model. This reduces the computational and storage resource overhead of dynamic tokens and improves the efficiency of computer information processing.

[0073] Figure 3 A block diagram illustrating some embodiments of the information processing device of the present disclosure.

[0074] like Figure 3 As shown, the information processing device 3 includes: a vector calculation module 31, which is used to use a machine learning model to calculate the query vector and key vector of each labeled data in the multiple labeled data included in the information to be processed; an attention calculation module 32, which is used to calculate the first attention weight of each labeled data based on partial features in the query vector and partial features in the key vector; a processing module 33, which is used to determine the processing result of the information to be processed based on the first attention weight.

[0075] In some embodiments, the processing module 33 determines key tag data among the multiple tag data based on the first attention weight; and determines a processing result using a machine learning model based on the key tag data.

[0076] In some embodiments, the processing module 33 determines at least one labeled data with the largest first attention weight as key labeled data.

[0077] In some embodiments, the attention calculation module 32 calculates the second attention weight of each tagged data based on the query vector of each tagged data and the key vector of the key tagged data; the processing module determines the processing result of the information to be processed using the machine learning model based on the second attention weight.

[0078] In some embodiments, multiple processing units in a machine learning model process information to be processed based on the same key tag data, and the multiple processing units are arranged adjacent to each other in the machine learning model.

[0079] In some embodiments, the multiple processing units are multiple attention heads arranged adjacent to each other in a machine learning model, and / or multiple layers arranged adjacent to each other in a machine learning model.

[0080] In some embodiments, the processing unit includes a first processing unit and a second processing unit, and the processing module uses the second processing unit to determine the processing result according to the key tag data determined in the first processing unit.

[0081] In some embodiments, a predetermined number of adjacent query vectors correspond to the same key tag data.

[0082] In some embodiments, a preset number of adjacent query vectors include a first query vector and a second query vector. The processing module determines the key tag data corresponding to the first query vector based on the first attention weight corresponding to the first query vector, and determines the key tag data corresponding to the first query vector as the key tag data corresponding to the second query vector.

[0083] Figure 4 A block diagram showing some other embodiments of the information processing device of the present disclosure.

[0084] like Figure 4 As shown, memory 41 is used to store one or more computer-readable instructions. Memory 41 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 111 may, for example, store an operating system, application programs, a boot loader, a database, and other programs, as well as various application programs and various data.

[0085] The processor 42 is configured to execute computer-readable instructions to implement the information processing method described in any of the aforementioned embodiments or the information processing method described in any of the aforementioned embodiments. The specific implementation of each step of the information processing method can be found in the aforementioned embodiments, and the repeated parts are not repeated here.

[0086] The processor 42 can be configured to execute the steps of any of the above embodiments. The processor 42 can be embodied as various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be an X86 or ARM architecture, etc.

[0087] The processor 42 and the memory 41 can communicate with each other directly or indirectly. For example, the processor 42 and the memory 41 can communicate via a network. The network can include a wireless network, a wired network, and / or any combination of wireless and wired networks. The processor 42 and the memory 41 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0088] It should be noted that Figure 4 The components of the information processing device 4 shown are merely exemplary and non-limiting. The information processing device 4 may also have other components according to actual application requirements. The processor 42 may control the other components in the information processing device 4 to perform desired functions.

[0089] The information processing device 4 may be implemented by software, firmware and / or hardware, and may be integrated into a device installed with relevant application programs.

[0090] Figure 5A block diagram showing still further embodiments of the information processing device of the present disclosure.

[0091] Figure 5 The information processing device 5 shown may be a computer system with a dedicated hardware structure, which can execute corresponding functions when a relevant application program is installed.

[0092] Information processing devices include but are not limited to mobile terminals such as smart phones, laptops, personal digital assistants (PDAs), tablet personal computers (Tablet PCs), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., as well as fixed terminals such as digital televisions, desktop computers, etc.

[0093] like Figure 5 As shown, the central processing unit (CPU) 51 performs various processes according to the program stored in the read-only memory (ROM) 52 or the program loaded from the storage part 58 to the random access memory (RAM) 53. In the RAM 53, data required when the CPU 51 performs various processes is stored as needed. The central processing unit is only exemplary, and it can also be other types of processors, such as the various processors described above. The ROM 52, RAM 53 and the storage part 58 can be various forms of computer-readable storage media. It should be noted that although Figure 5 ROM 52, RAM 53 and storage portion 58 are shown separately in FIG, but one or more of them may be combined or located in the same or different memory or storage modules.

[0094] The CPU 51, the ROM 52, and the RAM 53 are connected to one another via a bus 54. To the bus 54, an input / output interface 55 is also connected.

[0095] The following components are connected to the input / output interface 55: an input portion 56 such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 57 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 58 including a hard disk, a magnetic tape, etc.; and a communication portion 59 including a network interface card such as a LAN card, a modem, etc. The communication portion 59 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Figure 5 The parts of the information processing device 5 shown in the figure communicate via the bus 54, but they can also communicate via a network or other means, where the network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0096] A drive 510 is also connected to the input / output interface 55 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 510 as needed so that a computer program read therefrom is installed in the storage section 58 as needed.

[0097] When the series of processes described above is implemented by software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 511 .

[0098] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product that, when the computer program product is run on a computer, causes the computer to implement the information processing method described in any of the aforementioned embodiments. The computer program product includes computer instructions carried on a computer-readable medium, containing program code for executing the information processing method shown in the flowchart. In such an embodiment, the computer instructions can be downloaded and installed from the network through the communication part 59, or installed from the storage part 58, or installed from the ROM 52. When the computer program is executed by the CPU 51, the information processing method of the embodiment of the present disclosure is executed.

[0099] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device or for use in conjunction with an instruction execution system, apparatus, or device.

[0100] The computer readable medium may be a computer readable storage medium, or a computer readable signal medium, or any combination of the two.

[0101] Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device. Computer instructions are stored on the computer-readable storage medium, and when the instructions are executed by the processor, the information processing method described in any of the aforementioned embodiments is implemented.

[0102] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0103] The computer-readable medium may be included in the information processing device, or may exist independently without being incorporated into the information processing device.

[0104] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the information processing method described in any of the aforementioned embodiments. For example, the instructions may be embodied as computer program codes.

[0105] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure can be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] The functions described above may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0108] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An information processing method, comprising: Calculate, using a machine learning model, a query vector and a key vector for each of a plurality of labeled data included in the information to be processed; Calculating a first attention weight for each labeled data according to some features in the query vector and some features in the key vector; determining a processing result of the information to be processed according to the first attention weight; Wherein, determining the processing result of the information to be processed according to the first attention weight includes: determining key labeled data among the plurality of labeled data according to the first attention weight; Determining the processing result using the machine learning model based on the key tag data; Wherein, a preset number of adjacent query vectors correspond to the same key tag data, and the preset number of adjacent query vectors include a first query vector and a second query vector; The determining, according to the first attention weight, key tag data among the plurality of tag data comprises: determining key tag data corresponding to the first query vector according to a first attention weight corresponding to the first query vector; The key tag data corresponding to the first query vector is determined as the key tag data corresponding to the second query vector.

2. The information processing method according to claim 1, wherein: The determining, according to the first attention weight, key tag data among the plurality of tag data comprises: At least one labeled data with the largest first attention weight is determined as the key labeled data.

3. The information processing method according to claim 1, wherein: Determining a processing result of the information to be processed according to the first attention weight includes: Calculating a second attention weight for each of the labeled data according to the query vector of each of the labeled data and the key vector of the key labeled data; According to the second attention weight, the machine learning model is used to determine the processing result of the information to be processed.

4. The information processing method according to any one of claims 1 to 3, wherein: The multiple processing units in the machine learning model process the information to be processed according to the same key tag data, and the multiple processing units are arranged adjacent to each other in the machine learning model.

5. The information processing method according to claim 4, wherein: The multiple processing units are multiple attention heads arranged adjacent to each other in the machine learning model, and / or multiple layers arranged adjacent to each other in the machine learning model.

6. The information processing method according to claim 4, wherein: The processing unit includes a first processing unit and a second processing unit, Determining the processing result by using the machine learning model according to the key tag data includes: The second processing unit is used to determine the processing result according to the key tag data determined in the first processing unit.

7. An information processing device comprising: A vector calculation module, configured to calculate a query vector and a key vector for each of a plurality of labeled data included in the information to be processed using a machine learning model; an attention calculation module, configured to calculate a first attention weight of each labeled data according to some features in the query vector and some features in the key vector; a processing module, configured to determine a processing result of the information to be processed according to the first attention weight; The processing module is specifically configured to determine key tag data among the plurality of tag data according to the first attention weight; and determine the processing result using the machine learning model according to the key tag data; Wherein, a preset number of adjacent query vectors correspond to the same key tag data, and the preset number of adjacent query vectors include a first query vector and a second query vector; The processing module is specifically configured to determine key tag data corresponding to the first query vector based on a first attention weight corresponding to the first query vector; and determine the key tag data corresponding to the first query vector as the key tag data corresponding to the second query vector.

8. An information processing device comprising: Memory; and A processor coupled to the memory, wherein the processor is configured to execute the information processing method according to any one of claims 1 to 6 based on instructions stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the information processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the information processing method according to any one of claims 1 to 6.

Images