Sequence data feature extraction method and system and storage medium

Abstract

The embodiment of the invention provides a feature extraction method and system for sequence data and a storage medium. The method comprises the steps that target sequence data are obtained, and the target sequence data comprise a plurality of elements; determining a plurality of subsequences in one-to-one correspondence with a plurality of target expert networks in the target sequence data, the plurality of target expert networks being at least a part of the plurality of alternative expert networks, and each subsequence comprising a part of elements in the target sequence data, the relevancy between the elements in each sub-sequence and the corresponding target expert network meets a target condition, and the relevancy represents the suitable processing degree of the expert network to the elements; processing the plurality of subsequences by using computing resources corresponding to the plurality of target expert networks to obtain feature sub-results corresponding to the plurality of subsequences; and determining feature representation corresponding to the target sequence data based on the feature sub-results corresponding to the plurality of subsequences.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system and storage medium for extracting features from sequence data. Background Art

[0002] At present, Natural Language Processing (NLP) and other machine learning fields are developing rapidly. As a core technology in the field of natural language processing, the attention mechanism can simulate the process of human attention focusing on the most critical part of information, helping the model to effectively allocate computing resources when processing long sequence data and focus on the most important part of the input information for predicting the results. Among them, the Transformer model has become a standard technology because of its good performance in a variety of complex tasks. With the growth of model size and the increase in the amount of data processed, the model performance has also been significantly improved.

[0003] However, despite the success of existing Transformer models in multiple tasks, they still face challenges in computing and memory resources when processing large sequence data. Standard self-attention mechanisms need to consider all interactions between each element in the sequence, resulting in their computational complexity and memory requirements growing with the square of the sequence length, which limits their use in application scenarios with long sequences or limited resources.

[0004] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0005] The present specification provides a method, system and storage medium for extracting features from sequence data, which can reduce video memory usage without reducing feature expression capabilities when extracting features from sequence data.

[0006] In a first aspect, the present specification provides a method for extracting features from sequence data, which is applied to a computing system in which a plurality of candidate expert networks are deployed, wherein different candidate expert networks are deployed on different computing resources in the computing system, the method comprising: obtaining target sequence data, wherein the target sequence data comprises a plurality of elements; determining a plurality of subsequences in the target sequence data corresponding one-to-one to a plurality of target expert networks, wherein the plurality of target expert networks are at least a part of the plurality of candidate expert networks, each subsequence comprises a portion of the elements in the target sequence data, and a correlation between the elements in each subsequence and its corresponding target expert network satisfies a target condition, wherein the correlation characterizes a degree of suitability of the expert network for processing the elements; processing the plurality of subsequences using the computing resources corresponding to the plurality of target expert networks to obtain feature subresults corresponding to each of the plurality of subsequences; and determining a feature representation corresponding to the target sequence data based on the feature subresults corresponding to each of the plurality of subsequences.

[0007] In some embodiments, each element in the target sequence data corresponds to an element representation vector, and the multiple target expert networks respectively correspond to expert representation vectors, and determining multiple subsequences in the target sequence data that correspond one-to-one to the multiple target expert networks includes: determining the correlation between each target expert network in the multiple target expert networks and each element in the target sequence data based on the expert representation vectors of each target expert network and the element representation vectors corresponding to each element in the target sequence data; and determining the multiple subsequences in the target sequence data that correspond one-to-one to the multiple target expert networks based on the correlation between the multiple target expert networks and each element in the target sequence data.

[0008] In some embodiments, based on the correlation between the multiple target expert networks and each element in the target sequence data, the multiple subsequences corresponding to the multiple target expert networks are determined in the target sequence data, including: for each target expert network in the multiple target expert networks, based on the correlation between the target expert network and each element in the target sequence data, multiple target elements are determined, and subsequences corresponding to the target expert network are generated based on the multiple target elements, wherein the multiple target elements are the top K elements in the target sequence data with the highest correlation with the target expert network, or are elements in the target sequence data whose correlation with the target expert network is in a specified interval.

[0009] In some embodiments, based on the correlation between the multiple target expert networks and each element in the target sequence data, determining the multiple subsequences corresponding one-to-one to the multiple target expert networks in the target sequence data includes: for each element in the target sequence data, based on the correlation between the element and the multiple target expert networks, determining at least one target expert network suitable for processing the element in the multiple target expert networks; and based on the correlation between each element in the target sequence data and the at least one target expert network corresponding to it, determining the multiple subsequences corresponding one-to-one to the multiple target expert networks in the target sequence data.

[0010] In some embodiments, the multiple subsequences corresponding to the multiple target expert networks are determined based on the correlation between each element in the target sequence data and the corresponding at least one target expert network, including: for each target expert network in the multiple target expert networks, based on the correlation between the target expert network and the elements suitable for processing, multiple target elements are determined among the elements suitable for processing by the target expert network, and subsequences corresponding to the target expert network are generated based on the multiple target elements, wherein the multiple target elements are the multiple elements in the target sequence data with the highest correlation with the target expert network, or the multiple elements in the target sequence data with the target expert network in a specified interval.

[0011] In some embodiments, at least one target expert network suitable for processing the element includes: the top N expert networks with the highest correlation with the element among the multiple target expert networks, where N is an integer greater than or equal to 1, or an expert network among the multiple target expert networks whose correlation with the element is greater than or equal to a preset threshold.

[0012] In some embodiments, the multiple target expert networks are some expert networks among the multiple alternative expert networks, and the method also includes: determining the correlation between the target sequence data and the multiple alternative expert networks; and based on the correlation between the target sequence data and the multiple alternative expert networks, determining the multiple alternative expert networks among the multiple alternative expert networks with the highest correlation with the target sequence data as the multiple target expert networks.

[0013] In some embodiments, after determining the multiple subsequences corresponding one-to-one to the multiple target expert networks, if there are elements to be processed in the target sequence data that are not included in any subsequence, the method further includes: adding the elements to be processed to the subsequence containing the least number of elements among the multiple subsequences; or, creating a new target expert network in the computing system, and generating a subsequence corresponding to the new target expert network based on the elements to be processed; or, selecting a new target expert network from the multiple alternative expert networks, and generating a subsequence corresponding to the new target expert network based on the elements to be processed.

[0014] In some embodiments, the feature sub-result includes a feature vector corresponding to an element in the target sequence data, and determining the feature representation corresponding to the target sequence data based on the feature sub-results corresponding to each of the multiple sub-sequences includes: for each element in the target sequence data, using the correlation between the element and the multiple target expert networks as a weight, performing weighted summation of the feature vectors corresponding to the element in the multiple feature sub-results to determine the feature result of the element; and determining the feature representation corresponding to the target sequence data based on the feature results of each element in the target sequence data.

[0015] In some embodiments, the processing of the multiple subsequences using the computing resources corresponding to the multiple target expert networks includes: simultaneously inputting the multiple subsequences into their respective corresponding target expert networks, so as to perform parallel processing of the multiple subsequences through the computing resources corresponding to the multiple target expert networks.

[0016] In a second aspect, the present specification also provides a feature extraction system for sequence data, comprising: at least one storage medium storing at least one instruction set for feature extraction; and at least one processor communicatively connected to the at least one storage medium, wherein when the feature extraction system is running, the at least one processor reads the at least one instruction set and executes the feature extraction method as described in any one of the first aspects according to the instructions of the at least one instruction set.

[0017] In a third aspect, the present specification also provides a computer-readable non-volatile storage medium, wherein at least one instruction set is stored in the computer-readable non-volatile storage medium, and when the at least one instruction set is executed by at least one processor, the feature extraction method for sequence data as described in any one of the first aspects is implemented.

[0018] It can be seen from the above technical solutions that the feature extraction method of sequence data provided in this specification determines multiple subsequences corresponding to multiple target expert networks in the obtained target sequence data, and uses the computing resources corresponding to the multiple target expert networks to process the multiple subsequences to obtain the feature sub-results corresponding to each of the multiple subsequences; and based on the feature sub-results corresponding to each of the multiple subsequences, determines the feature representation corresponding to the target sequence data. In the solution provided in this specification, multiple target expert networks are used to process the target sequence data, each target expert network only processes the corresponding subsequence, and the length of the sequence processed by each target expert network is significantly reduced, avoiding the waste of computing resources, improving computing efficiency and reducing video memory requirements. Furthermore, the correlation between the elements in each subsequence and its corresponding target expert network meets the target condition, and the correlation is the degree of suitability of the expert network to process the elements. Therefore, for each target expert network, each target expert network only processes part of the data related to itself. On the basis of not losing feature information, this division of expertise enables the model as a whole to better adapt to different inputs and improve the generalization ability of the model. In general, the feature extraction method of sequence data provided in this specification can reduce video memory occupancy without reducing the feature expression ability.

[0019] Other functions of feature extraction of sequence data provided in this specification will be partially listed in the following description. The creative aspects of feature extraction of sequence data provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of an application scenario of feature extraction of sequence data provided according to an embodiment of this specification is shown;

[0022] Figure 2 A hardware structure diagram of feature extraction of sequence data provided according to an embodiment of this specification is shown;

[0023] Figure 3 A flow chart of a method for extracting features from sequence data provided in accordance with an embodiment of this specification is shown;

[0024] Figure 4A schematic diagram of a feature extraction process for sequence data provided according to an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0025] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0026] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0027] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0028] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0029] The following is an introduction to the application scenarios of this manual.

[0030] The feature extraction scheme for sequence data provided in this specification can be used for application scenarios of feature extraction from sequence data. It should be noted that the embodiments of this specification do not limit the specific feature extraction field, that is, it can be used for feature extraction in any field. In addition, it is also worth further explaining that the feature extraction scheme for sequence data provided in this specification can be used as one of the layers of a large language model to help the large model understand the semantics, grammar and emotions of the language, and support machine translation, text summarization, sentiment analysis, speech recognition, etc. In some embodiments, the feature extraction scheme for sequence data provided in this specification can also be used as a method for model construction, and task processing for scenarios such as feature extraction can be performed separately. In more specific application scenarios, the feature extraction scheme for sequence data provided in this specification can be applied to scenarios including but not limited to: extracting the main content of a long document or long speech input by the user to generate a concise summary; generating a keyword list containing key information by identifying important words (such as high-frequency words, subject words) in a given text or a given speech; automatically extracting important words from the text to help information retrieval and search engine optimization; extracting features from user questions and candidate answers to match the most relevant answers.

[0031] Those skilled in the art should understand that the above-listed recommended scenarios are only part of the applicable scenarios of the embodiments of this specification. In addition to the above-listed scenarios, the embodiments of this specification can also be used in other recommended scenarios, which are not described one by one in this specification.

[0032] Figure 1 FIG. 1 is a schematic diagram showing an application scenario of feature extraction of sequence data provided in accordance with an embodiment of this specification. Figure 1 As shown, the application scenario 100 may include a target user 110 , a client 120 , a server 130 , and a network 140 .

[0033] The target user 110 may be a user who triggers feature extraction of sequence data, and the target user 110 may perform a feature extraction operation on the sequence data on the client 120 .

[0034] The client 120 may be a device that extracts features from target sequence data in response to a feature extraction operation of the sequence data of the target user 110. In some embodiments, the method for extracting features from sequence data may be executed on the client 120. At this time, the client 120 may store data or instructions for executing the method for extracting features from sequence data described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the client 120 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work.

[0035] In some embodiments, the client 120 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, or the like, or any combination thereof. In some embodiments, the virtual reality device or the augmented reality device may include a virtual reality helmet, virtual reality glasses, a virtual reality patch, an augmented reality helmet, augmented reality glasses, an augmented reality patch, or the like, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, a head mounted display, VR, or the like. In some embodiments, the built-in device in the motor vehicle may include an onboard computer, an onboard TV, or the like. In some embodiments, the client 120 may include a text or voice acquisition device for acquiring text or voice information in the target space, thereby acquiring target sequence data.

[0036] In some embodiments, the client 120 may be installed with one or more application programs (APPs). The APP can provide the target user 110 with the ability and interface to interact with the outside world through the network 140. The APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc. In some embodiments, a target APP may be installed on the client 120. The target APP can collect text or voice information in the target space for the client 120, thereby obtaining target sequence data. In some embodiments, the target user 110 can also trigger a feature extraction request for sequence data through the target APP. The target APP can respond to the feature extraction request for the sequence data and execute the method for feature extraction of sequence data described in this specification.

[0037] The server 130 may be a server that provides various services, such as a background server that provides support for feature extraction of sequence data collected on the client 120. In some embodiments, the method for feature extraction of sequence data may be executed on the server 130. At this time, the server 130 may store data or instructions for executing the method for feature extraction of sequence data described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the server 130 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work. The server 130 may be connected to multiple clients 120 for communication, and receive data sent by the client 120.

[0038] The network 140 is used to provide a medium for communication connection between the client 120 and the server 130. The network 140 can facilitate the exchange of information or data. Figure 1 As shown, the client 120 and the server 130 can be connected to the network 140 and transmit information or data to each other through the network 140. In some embodiments, the network 140 can be any type of wired or wireless network, or a combination thereof. For example, the network 140 can include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, or a similar network. In some embodiments, the network 140 can include one or more network accesses. For example, the network 140 can include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of the client 120 and the server 130 can be connected to the network 140 to exchange data or information.

[0039] It should be understood that Figure 1 The number of clients 120, servers 130 and networks 140 in the embodiment is only illustrative. Any number of clients 120, servers 130 and networks 140 may be provided according to implementation requirements.

[0040] It should be noted that the method for extracting features from sequence data may be completely executed on the client 120 , may be completely executed on the server 130 , or may be partially executed on the client 120 and partially executed on the server 130 .

[0041] It should be noted that the user data obtained in this manual has been authorized by the user and does not involve user privacy.

[0042] Figure 2FIG. 2 shows a hardware structure diagram of a computing device 200 provided according to an embodiment of the present specification. The computing device 200 can be used as Figure 1 For example, the client 120 or the server 130 executes the method for extracting features from sequence data described in this specification. The embodiments of this specification also provide a system for extracting features from sequence data (hereinafter referred to as a computing system), which can execute the method for extracting features from sequence data described in this specification. Figure 2 The computing device 200 shown can be used as the above computing system. In some embodiments, when the computing system adopts a cluster architecture, Figure 2 The illustrated computing device 200 may also be implemented as a single computing device in a cluster architecture.

[0043] like Figure 2 As shown, computing device 200 may include at least one storage medium 230 and at least one processor 220. In some embodiments, computing device 200 may also include communication port 250 and internal communication bus 210. Computing device 200 may also include I / O component 260.

[0044] The internal communication bus 210 can connect different system components. For example, the internal communication bus 210 can connect the storage medium 230, the processor 220, the communication port 250, and the I / O component 260, etc.

[0045] I / O components 260 support input / output between computing device 200 and other components.

[0046] The communication port 250 is used for data communication between the computing device 200 and the outside world. For example, the communication port 250 can be used for data communication between the computing device 200 and the network 140. The communication port 250 can be a wired communication port or a wireless communication port.

[0047] The storage medium 230 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 235. The storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set may include computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, and the like.

[0048] At least one processor 220 may be in communication with at least one storage medium 230. When the computing device 200 is running, at least one processor 220 reads the at least one instruction set and executes the method for feature extraction of sequence data provided in this specification according to the instructions of the at least one instruction set. The processor 220 may execute the steps included in the method for feature extraction of sequence data. The processor 220 may be in the form of one or more processors. In some embodiments, the processor 220 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0049] For illustration purposes only, the computing device 200 in the figure shows only one processor 220. However, it should be noted that the computing device 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or by multiple processors in combination. For example, if the processor 220 of the computing device 200 is described in this specification as performing step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 220 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0050] As mentioned above, the existing task of feature extraction of sequence data is usually performed by the Transformer model. The standard attention mechanism is slightly insufficient in processing long sequences or resource-limited application scenarios, and both computing resources and video memory resources are challenged. In order to solve the above problems, a variety of variants of self-attention mechanisms have been proposed, such as sparse attention, local attention, and low-rank approximation. Although these variants reduce the computing and video memory pressure to a certain extent, they still have some limitations. For example, sparse attention may miss important interactions, such as splitting the sequence into small blocks or using hash functions to approximate the attention distribution through sparse attention's "Locality Sensitive Hashing Attention, LSH" and "Block Sparse Attention", thereby avoiding full connection calculations for all sequence positions. Although this method reduces the amount of calculation, these methods often introduce approximation errors. Due to the omission of attention between different segments, it will affect the performance of the final model, especially when dealing with long-distance dependencies. Similarly, local attention will also limit the capture of global information. In order to improve efficiency, low-rank approximation in multi-head attention may sacrifice the expressiveness of the model. For example, shared parameters may limit the model's ability to capture complex features, but increase the approximation error. It can be seen that although the above method reduces the computational burden to a certain extent, for large-scale models, the saved resources may be relatively limited and still insufficient to cope with extreme-scale data processing needs, or may introduce new problems.

[0051] Based on the above analysis, Figure 3 FIG. 1 is a flow chart of a method P300 for extracting features from sequence data according to an embodiment of the present specification. Figure 3 The feature extraction method P300 for sequence data is described in detail.

[0052] The computing system can execute the sequence data feature extraction method P300 provided in this specification. The computing system refers to the entire computing environment, including hardware, software, and various resources required to perform the sequence data feature extraction task. In some embodiments, the computing system can correspond to a single Figure 2 In some embodiments, the computing system may correspond to a computing device such as Figure 2 The computing cluster composed of multiple computing devices shown. For example, the computing system can be composed of multiple servers, distributed computing nodes or cloud computing platforms.

[0053] Multiple candidate expert networks are deployed in the computing system. Each candidate expert network can be used to specifically process a subsequence composed of some elements of the target sequence data. Different candidate expert networks are deployed on different computing resources in the computing system, so that the computing system can support the serial and parallel operation of multiple candidate expert networks. Computing resources are the hardware and software resources required for the expert network to perform data processing, computing and storage tasks, which may include CPU resources, GPU resources, memory resources, video memory resources, storage resources, etc., to ensure that the expert network has sufficient processing power and storage space to process the corresponding subsequences. For each expert network, being assigned different computing resources enables different expert networks to work independently and in parallel, avoiding resource conflicts. The computing system can extract features from sequence data in parallel and efficiently, and effectively cope with the processing tasks of large amounts of data.

[0054] like Figure 3 As shown, method P300 may include:

[0055] S310: Obtain target sequence data, where the target sequence data includes multiple elements.

[0056] In some embodiments of the present specification, the target sequence data can be generated based on text, audio or image, that is, it can be generated based on text data, such as sentences, paragraphs, articles, etc.; it can also be generated based on audio data, such as voice data, etc.; it can be generated based on image data, such as two-dimensional images or continuous image frames or video frames; of course, it can also be based on some composition data through time series, such as continuous sampling time series, etc., which is not limited in the present specification. The above data can be converted into sequence data in vector form for subsequent processing. The method for converting data into vectors can adopt any existing supported conversion method, which is not limited in the present specification.

[0057] In some embodiments of the present specification, the multiple elements included in the target sequence data may be divided into granularity based on the elements in the source data (text, audio or image), or may be divided into granularity based on the vectors in the target sequence. Those skilled in the art will appreciate that the reference contents of the multiple elements included in the target sequence data may be different in different usage scenarios.

[0058] S320: Determine a plurality of subsequences corresponding one-to-one to a plurality of target expert networks in the target sequence data, wherein the plurality of target expert networks are at least part of the plurality of candidate expert networks, each subsequence includes a portion of elements in the target sequence data, and a correlation between the elements in each subsequence and its corresponding target expert network satisfies a target condition, and the correlation characterizes a degree of suitability of the expert network for processing the elements.

[0059] In some embodiments of the present specification, different candidate expert networks are deployed on different computing resources in the computing system, and each candidate expert network is used to specifically process a subsequence consisting of some elements of the target sequence data. Before extracting features from the sequence data, the computing system creates an expandable group of candidate expert networks. Through continuous training, each expert network can independently learn different subsequences in the sequence data. Moreover, in actual use, the computing system can further train the expert network through actual data, so that each expert network can focus more on the elements most relevant to it, so that the computing efficiency of each expert network is higher than the standard full self-attention calculation.

[0060] In some embodiments of the present specification, the computing system may dynamically allocate elements in a subsequence based on the performance of the model or other real-time indicators, so that the model can more flexibly expand or reduce its capacity to adapt to different tasks and sequence data sets.

[0061] In some embodiments of the present specification, each subsequence includes some elements in the target sequence data. Elements of different subsequences may or may not overlap partially. In the case of partial overlap, it means that the same element can also enter the subsequences corresponding to multiple target expert networks at the same time, provided that the correlation between the element and its corresponding target expert network meets the target condition. The model can be further optimized based on the partial overlap of subsequences to avoid information loss due to truncation.

[0062] In some embodiments of the present specification, the target expert network is at least part of the multiple candidate expert networks. Such an architecture supports the scalability and modularity of the model, allowing the system to flexibly add or modify expert networks to adapt to different processing requirements or updated technologies. Different sequence data can share the same expert network library. Furthermore, the computing system can also classify the candidate expert networks and make more detailed divisions according to different processing requirements, so that the processing of the expert network is more targeted and the processing quality and accuracy are improved.

[0063] S330: Processing the multiple subsequences using computing resources corresponding to the multiple target expert networks to obtain feature sub-results corresponding to each of the multiple subsequences.

[0064] In some embodiments of the present specification, the computing system sends the subsequence corresponding to each target expert network to its network for processing. The computing resources corresponding to the multiple target expert networks process the multiple subsequences, and different expert networks have parallel processing capabilities, which effectively utilizes computing resources and improves the processing efficiency of the computing system. For each subsequence, a corresponding feature subresult can also be obtained.

[0065] In some embodiments of the present specification, the feature sub-result obtained after feature extraction of each subsequence represents key information in the subsequence, and can also be understood as feature information extracted from each subsequence.

[0066] S340: Determine a feature representation corresponding to the target sequence data based on the feature sub-results corresponding to each of the multiple sub-sequences.

[0067] In some embodiments of the present specification, a feature representation including main characteristics and information of the target sequence data is formed by integrating the feature sub-results corresponding to each sub-sequence.

[0068] In summary, the feature extraction method of sequence data provided in this specification determines multiple subsequences corresponding to multiple target expert networks in the obtained target sequence data, and uses the computing resources corresponding to the multiple target expert networks to process the multiple subsequences to obtain the feature sub-results corresponding to each of the multiple subsequences; and based on the feature sub-results corresponding to each of the multiple subsequences, determines the feature representation corresponding to the target sequence data. In the scheme provided in this specification, multiple target expert networks are used to process the target sequence data, each target expert network only processes the corresponding subsequence, and the length of the sequence processed by each target expert network is significantly reduced, avoiding the waste of computing resources, improving computing efficiency and reducing video memory requirements. Furthermore, the correlation between the elements in each subsequence and its corresponding target expert network meets the target condition. The correlation is the degree of suitability of the expert network to process the elements. Therefore, for each target expert network, each target expert network only processes part of the data related to itself. On the basis of not losing feature information, this division of expertise enables the model as a whole to better adapt to different inputs and improve the generalization ability of the model. In general, the feature extraction method of sequence data provided in this specification can reduce video memory occupancy without reducing the feature expression ability.

[0069] In some embodiments, the computing system may determine a plurality of subsequences corresponding to a plurality of target expert networks in the target sequence data in the following manner: based on the expert representation vectors of the plurality of target expert networks and the element representation vectors corresponding to the elements in the target sequence data, determine the correlation between each of the plurality of target expert networks and each element in the target sequence data; and based on the correlation between the plurality of target expert networks and each element in the target sequence data, determine the plurality of subsequences corresponding to the plurality of target expert networks in the target sequence data.

[0070] In some embodiments, different expert networks can be models of different structures, feature extraction modules of specific types, or sub-models specially trained to process specific features, and each expert network can be regarded as a small model that can capture certain specific aspects of the data. For example, the expert network can be a model of different structures, such as a convolutional neural network (CNN) model, a large language model (LLM) network model, a Transformer model, or a recurrent neural network (RNN) model. Each expert network can be a model of the above-mentioned general network or a combination of multiple models. For example, expert network 1 is a CNN model, expert network 2 is an LLM network model, and expert network 3 is a combination of a Transformer model and an RNN model. Different expert networks can be obtained through specific data sets or task-specific training to better process input data of certain specific patterns. Due to the different network structures of different expert networks and the different training data sets, the elements that they are good at processing are also different. The differences between expert networks can be reflected by expert representation vectors. For example, the CNN model is good at extracting local features, the LLM is more prominent in understanding complex semantic relationships and contextual dependencies, the Transformer can dynamically focus on the relationship between each position in the sequence through the self-attention mechanism, and the RNN is good at processing short-term dependent sequence data, etc. Therefore, through the structural and functional differences of different expert networks themselves, each expert network can focus on the elements that meet the corresponding relevance conditions. Each expert network can maximize its capabilities while generating more expressive features. In the actual training or application process, the composition of different expert networks can be adjusted based on different application scenarios to adapt to the diverse characteristics of complex sequence data.

[0071] In some embodiments, the element characterization vector can be obtained through the embedding layer of the model or the preprocessing model. The expert characterization vector can be fixed in the computing system or dynamically generated in the process of processing different sequence data. In some embodiments, the measure of similarity can be obtained by the dot product between vectors, cosine similarity, or a score based on an attention mechanism. In some embodiments, a correlation score vector with the target expert network as the partition dimension can be generated by summarizing the similarity calculation results of each target expert network, and these scores represent the correlation between the input element and each target expert network. Alternatively, another correlation score vector with the element as the partition dimension can be generated by summarizing the similarity calculation results of each element, and these scores also represent the correlation between the input element and each expert.

[0072] In some embodiments, a gating network can be used to learn and determine the correlation between each target expert network and each element in the target sequence data. The gating network is a fully connected layer, whose output is normalized by a softmax function, and can directly generate a probability distribution related to all target expert networks for each sequence element, or generate a probability distribution related to all elements for each target expert network.

[0073] For ease of understanding, the process of determining the correlation between each target expert network in the multiple target expert networks and each element in the target sequence data is described below by way of example.

[0074] Please refer to Figure 4 , assuming that there are three target expert networks, namely W1, W2 and W3, and the target sequence data S contains 5 elements [s1, s2, s3, s4, s5]:

[0075] The computing system inputs the target sequence data and the target expert network into the gating network respectively. The gating network can generate a probability distribution related to all target expert networks for each sequence element. Taking the first line below as an example, P(s1) indicates that the correlation between element s1 and expert network W1, expert network W2, and expert network W3 is 0.3, 0.5, and 0.2 respectively.

[0076] P (s1) =[0.3,0.5,0.2];

[0077] P (s2) =[0.4,0.25,0.35];

[0078] P (s3) =[0.1,0.5,0.4];

[0079] P (s4) =[0.2,0.3,0.5];

[0080] P (s5) =[0.35,0.2,0.45].

[0081] It is worth noting that, in order to facilitate understanding of the current solution, the number of expert networks and the length of sequence data selected in the above embodiments are relatively small. In actual generation applications, the embodiments in this specification can handle feature extraction of long sequences.

[0082] In some embodiments, the computing system may determine the multiple subsequences corresponding one-to-one to the multiple target expert networks in the target sequence data by executing S321.

[0083] S321: For each target expert network among the multiple target expert networks, multiple target elements are determined based on the correlation between the target expert network and each element in the target sequence data, and a subsequence corresponding to the target expert network is generated based on the multiple target elements, wherein the multiple target elements are the first K elements in the target sequence data with the highest correlation with the target expert network, or the elements in the target sequence data whose correlation with the target expert network is in a specified interval.

[0084] In some embodiments, the target element can be directly used as a subsequence element, and those skilled in the art can formulate target element selection rules according to actual needs. In some embodiments of the present specification, the first K elements with the highest correlation with the target expert network in the target sequence data can be selected as target elements, and the K target elements can be directly used as subsequence elements; or elements in the target sequence data whose correlation with the target expert network is in a specified interval can be selected as target elements. Furthermore, the K value or specified interval range corresponding to each target expert network can be the same or different, that is, the target element length determined by each target expert network can be the same or different. Similarly, the subsequence length corresponding to each target expert network can be the same or different.

[0085] In some embodiments of this specification, the selection of Top-K or specified intervals ensures that each expert network can focus on the most important interactions. By selecting the subsequences that each expert network focuses on, the expert network can retain the most important information while avoiding long-distance dependencies that may be ignored by simple low-rank approximation. The strategies of the above embodiments can effectively reduce the information loss problem that may be caused by low-rank approximation. By focusing on the most important element interactions in the sequence data, the video memory required in the self-attention calculation process is greatly reduced, allowing the system to process longer sequences at a lower resource cost.

[0086] Please refer to Figure 4 , still taking the above three target expert networks and target sequence data S as an example, how to determine multiple subsequences corresponding to multiple target expert networks according to S321:

[0087] 1. Assume that for each expert network, K is 3, that is, for each expert network, the target element is the first three elements in the target sequence data with the highest correlation with the target expert network, then the subsequences w1, w2 and w3 corresponding to the three target expert networks are:

[0088] w1:[s1,s2,s5];

[0089] w2:[s1,s3,s4];

[0090] w3:[s3,s4,s5].

[0091] Taking the first row as an example, the subsequence w1 corresponding to the target expert network W1 contains the following elements: s1, s2, s5.

[0092] 2. Assume that for each expert network, the specified interval of correlation is [0.3-1]. That is, for each expert network, the target element is the element in the target sequence data whose correlation with the target expert network is [0.3-1]. Then the subsequences w1, w2 and w3 corresponding to the three target expert networks are:

[0093] w1:[s1,s2,s5];

[0094] w2:[s1,s3,s4];

[0095] w3:[s2,s3,s4,s5].

[0096] Taking the first row as an example, the subsequence w1 corresponding to the target expert network W1 contains the following elements: s1, s2, s5.

[0097] In some embodiments, the computing system may further determine the plurality of subsequences corresponding one-to-one to the plurality of target expert networks in the target sequence data by executing S322, including:

[0098] S322: For each element in the target sequence data, based on the correlation between the element and the multiple target expert networks, determine at least one target expert network suitable for processing the element in the multiple target expert networks; and based on the correlation between each element in the target sequence data and the corresponding at least one target expert network, determine in the target sequence data the multiple subsequences corresponding one-to-one to the multiple target expert networks.

[0099] In some embodiments, the most relevant expert networks for each element may be found first, and then the subsequence of each target expert network may be found. Furthermore, the number of target expert networks corresponding to each element may be the same or different, and the length of the subsequence corresponding to each target expert network may be the same or different.

[0100] Still taking the above three target expert networks and target sequence data S as an example, how to determine multiple subsequences corresponding to multiple target expert networks according to S322:

[0101] Assume that for each element, two target expert networks suitable for processing the element are selected from the target expert networks. Then, for each element, the selected target expert networks are:

[0102] For each element, the target expert networks selected are:

[0103] s1: [W1, W2];

[0104] s2: [W1, W3];

[0105] s3: [W2, W3];

[0106] s4: [W2, W3];

[0107] s5:[W1,W3].

[0108] In some embodiments, the computing system can also determine the multiple subsequences corresponding to the multiple target expert networks one by one based on S322 in the following manner, including: for each target expert network in the multiple target expert networks, based on the correlation between the target expert network and the elements suitable for processing, multiple target elements are determined among the elements suitable for processing by the target expert network, and subsequences corresponding to the target expert network are generated based on the multiple target elements, wherein the multiple target elements are the multiple elements in the target sequence data with the highest correlation with the target expert network, or the multiple elements in the target sequence data with the target expert network in a specified interval.

[0109] In some embodiments, at least one target expert network suitable for processing the element includes: the top N expert networks with the highest correlation with the element among the multiple target expert networks, where N is an integer greater than or equal to 1, or an expert network among the multiple target expert networks whose correlation with the element is greater than or equal to a preset threshold.

[0110] In some embodiments, firstly, based on selecting N target expert networks for each element, the target element and the corresponding subsequence are screened for the target expert network. It can be inferred that the target element can be directly used as a subsequence element. For each element, the corresponding N value can be the same or different, that is, the number of expert networks selected for each element can be the same or different.

[0111] In some embodiments, those skilled in the art can formulate target element selection rules according to actual needs. In some embodiments of the present specification, the first M elements in the target sequence data with the highest correlation with the target expert network can be selected as target elements, and the M target elements can be directly used as subsequence elements; or elements in the target sequence data with the correlation with the target expert network in a specified interval can be selected as target elements. Furthermore, the M value or specified interval range corresponding to each target expert network can be the same or different, that is, the target element length determined by each target expert network can be the same or different. Similarly, the subsequence length corresponding to each target expert network can be the same or different.

[0112] Still taking the above three target expert networks and target sequence data S as an example, how to determine the target expert network for each element selection according to S322 is explained:

[0113] 1. Based on the selection of two target expert networks for each element, assuming that for each expert network, M is 3, that is, for each expert network, the target element is the first three elements in the target sequence data with the highest correlation with the target expert network, then the subsequences w1, w2 and w3 corresponding to the three target expert networks are:

[0114] w1:[s1,s2,s5];

[0115] w2:[s1,s3,s4];

[0116] w3:[s3,s4,s5].

[0117] 2. Based on the selection of two target expert networks for each element, assuming that for each expert network, the specified interval of correlation is [0.3-1], that is, for each expert network, the target element is the element in the target sequence data whose correlation with the target expert network is [0.3-1]. Then the subsequences w1, w2 and w3 corresponding to the three target expert networks are:

[0118] w1:[s1,s2,s5];

[0119] w2:[s1,s3,s4];

[0120] w3:[s2,s3,s4,s5].

[0121] It is worth noting that although the results of selecting the subsequence corresponding to the target expert network under the corresponding rules in S321 and S322 are the same, this is due to the limitations of the data in the above embodiment. However, in actual application, technicians in this field can think of different selection methods that can produce different effects and results.

[0122] By executing S321, the target element can be directly selected for each expert network to determine the subsequence, reducing the multiple selection and screening processes, making the overall calculation process of the system simpler and more efficient, and reducing the calculation time and complexity. By executing S322, the corresponding relationship between the element and the expert is first selected, and then M elements are selected for each expert for processing, which helps to reasonably distribute the load of the expert network and avoid overloading of some expert networks.

[0123] In some embodiments, the computing system can also filter the target expert network from the alternative expert networks based on the correlation: the multiple target expert networks are some expert networks among the multiple alternative expert networks, and the method also includes: determining the correlation between the target sequence data and the multiple alternative expert networks; and based on the correlation between the target sequence data and the multiple alternative expert networks, determining the multiple alternative expert networks among the multiple alternative expert networks with the highest correlation with the target sequence data as the multiple target expert networks.

[0124] In some embodiments, an expandable alternative expert network group can also be created, and each expert network in the alternative expert network group can independently learn different feature subsets in the sequence to adapt to different tasks and data sets. For each target sequence data, the computing system can screen the expert networks in the alternative expert network so that the model can flexibly expand or reduce its capacity. Specifically, a score vector can be generated based on the correlation between the target sequence data and the multiple alternative expert networks, and each score corresponds to an alternative expert network, indicating the applicability of the alternative expert network to the current target sequence data. According to the score vector, multiple alternative expert networks with the highest scores are selected for activation, and the remaining alternative expert networks will not participate in the calculation, thereby realizing sparse activation of the expert network and reducing the amount of calculation. In some embodiments, the target expert network can also be screened through a gating network.

[0125] In the above embodiment, the number and complexity of the candidate expert networks and the target expert network are allowed to be flexibly adjusted. The screening of the expert network is combined with the screening of subsequences by each expert network, which provides the model with sufficient capacity to capture key patterns in the data. The integration and dynamic adjustment of multiple expert networks can effectively avoid the loss of key information.

[0126] If there are elements to be processed in the target sequence data that are not included in any subsequence, then during the feature extraction process, the information of these elements will be ignored, which may result in an incomplete description of the entire target sequence data. Therefore, in some embodiments, the computing system can also perform S323 to forcibly allocate the non-included elements after determining the multiple subsequences that correspond one-to-one to the multiple target expert networks, thereby ensuring that all data have a chance to be processed, thereby avoiding the loss of key information in the feature extraction stage, especially those details that may have a significant impact on the final result. Forcibly allocating the elements to be processed includes: S323: adding the elements to be processed to the subsequence containing the least number of elements among the multiple subsequences.

[0127] In some embodiments, by forcing the computing system to allocate elements to be processed, it is possible to ensure that all sequence elements have a chance to be processed, thereby avoiding the loss of key information in the feature extraction stage, while ensuring that all expert networks are fully utilized, avoiding the waste of resources of certain expert networks, balancing the workload of each expert network, and improving the overall resource utilization efficiency of the system.

[0128] Alternatively, the computing system executes S324 to create a new target expert network for the element to be processed.

[0129] S324: Create a new target expert network in the computing system, and generate a subsequence corresponding to the new target expert network based on the elements to be processed.

[0130] In some embodiments, the computing system can directly generate a new target expert network for the element to be processed based on the element representation vector of the element to be processed, or can select a general model as the new target expert network and process the subsequence composed of the elements to be processed based on the new target expert network.

[0131] Alternatively, the computing system executes S325 to select a new target expert network for the element to be processed.

[0132] S325: Select a new target expert network from the multiple candidate expert networks, and generate a subsequence corresponding to the new target expert network based on the element to be processed.

[0133] In some embodiments, if there are expert networks in the candidate expert networks that have not participated in the current target sequence data processing, the computing system selects a new target expert network from the remaining expert networks based on the correlation between the expert representation vector of the expert network and the element representation vector of the element to be processed. Alternatively, the computing system can forcibly assign a target expert network for processing.

[0134] By processing the elements to be processed that are not included in any subsequence in the above embodiment, the integrity of the sequence data can be ensured, and the correct flow of the gradient during back propagation can be determined, which is crucial for the effective update of the model parameters and can also improve the comprehensiveness of the feature representation. In some embodiments, the resource utilization of the expert network can also be improved. By fully processing the target sequence data, it helps the model cope with various complex input scenarios and generate more accurate and stable feature representations.

[0135] In some embodiments, the computing system can also determine the feature representation corresponding to the target sequence data based on the feature sub-results corresponding to each of the multiple sub-sequences in the following manner, including: for each element in the target sequence data, using the correlation between the element and the multiple target expert networks as a weight, performing weighted summation on the feature vectors corresponding to the element in the multiple feature sub-results to determine the feature result of the element; based on the feature results of each element in the target sequence data, determining the feature representation corresponding to the target sequence data.

[0136] In some embodiments, before the computing system performs weighted summation on the feature vectors corresponding to the elements in the multiple feature sub-results, the computing system normalizes the relevance of the elements to the multiple target expert networks on an element-by-element basis. The normalized relevance is used as a weight for weighted summation, and then each feature sub-result is scaled. Based on the normalized relevance, the weight imbalance problem caused by the different sizes of the original relevance values ​​can be eliminated, making the weighting process more stable, and also ensuring that in the weighted summation, the feature contribution of each expert network is dynamically adjusted to an appropriate level.

[0137] Please refer to Figure 4 , still taking the above three target expert networks and target sequence data S as an example, how to determine the feature representation corresponding to the target sequence data is explained:

[0138] Assume that the subsequences w1, w2 and w3 corresponding to the three target expert networks are:

[0139] w1:[s1,s2,s5];

[0140] w2:[s1,s3,s4];

[0141] w3:[s3,s4,s5].

[0142] The feature sub-results output by the three target expert networks are:

[0143] w1':[s11',s12',s15'];

[0144] w2':[s21',s23',s24'];

[0145] w3':[s33',s34',s35'].

[0146] The probability distribution of each sequence element associated with all target expert networks is as follows:

[0147] P (s1) =[0.3,0.5,0.2];

[0148] P (s2) =[0.4,0.25,0.35];

[0149] P (s3) =[0.1,0.5,0.4];

[0150] P (s4) =[0.2,0.3,0.5];

[0151] P (s5) =[0.35,0.2,0.45].

[0152] Performing weighted summation on the feature vectors corresponding to the elements in the multiple feature sub-results:

[0153] Taking element s1 as an example, s1 outputs corresponding sub-results in both the W1 network and the W2 network. Therefore, 0.3 and 0.5 are normalized to obtain 0.375 and 0.625; therefore, the final feature representation corresponding to s1 should be: 0.375*s11'+0.625*s21', and the same operation is performed on the sub-results corresponding to other elements.

[0154] The feature representation corresponding to the target sequence data is [s1', s2', s3', s4', s5'] = [0.375*s11'+0.625*s21', s12', 0.5556*s23'+0.4444*s33', 0.375*s24'+0.625*s34', 0.4375*s15'+0.5625*s35'].

[0155] In the above embodiment, the feature sub-results output by the target expert network are weighted and aggregated to obtain the final sequence representation. Although each expert network only focuses on the most relevant sub-sequence, the system can also capture global information by integrating the strategies of multiple target expert networks. Such dynamic routing and aggregation mechanisms enhance the model's processing capabilities for long sequence elements, making it more efficient and accurate when processing complex data structures. The embodiments in this specification provide a more balanced perspective on the features between local information by weighted summation of the feature sub-results.

[0156] In other embodiments, if there are elements to be processed that are not included in any subsequence, the computing system may not process the elements to be processed, but when finally determining the feature representation corresponding to the target sequence data: for each target element, use the correlation between the target element and the multiple target expert networks as a weight, perform weighted summation on the feature vectors corresponding to the element in the multiple feature sub-results, and determine the feature result of each target element; determine an initial feature representation based on the feature result of each target element and the element to be processed, wherein the feature vector corresponding to the element to be processed in the initial feature representation is 0; add the initial feature representation to the target sequence data to determine the feature representation corresponding to the target sequence data.

[0157] In some embodiments, the initial feature representation only includes feature results based on the output of the target element, but only using the initial feature representation as the feature representation of the entire target sequence data will result in the loss of some information in the feature extraction stage, especially the loss of detail information that has an important impact on the final result. Therefore, through the embodiments in this specification, the initial feature representation is added to the original target sequence data to determine the feature representation corresponding to the target sequence data, so that a more accurate feature representation can be obtained on the basis of retaining the original features.

[0158] In some embodiments, the computing system can also process the multiple subsequences using the computing resources corresponding to the multiple target expert networks in the following manner, including: simultaneously inputting the multiple subsequences into their respective corresponding target expert networks, so as to process the multiple subsequences in parallel through the computing resources corresponding to the multiple target expert networks.

[0159] In some embodiments, the computing resources corresponding to different expert networks can be distributed on different physical nodes. For example, the computing resources corresponding to expert network W1 are located on physical node 1, and the computing resources corresponding to expert network W2 are located on physical node 2. In some embodiments, the computing resources corresponding to different expert networks can also be distributed on the same physical node. For example, the computing resources corresponding to expert networks W1 and W2 are respectively part of the resources on physical node 1.

[0160] The above deployment methods can realize parallel processing of the corresponding subsequences by multiple target expert networks. During the parallel processing, each target expert network independently extracts features of the subsequences assigned to it, which can make full use of the parallel processing capabilities of modern hardware architectures and maximize computing efficiency and resource utilization through modular management. In some embodiments, resource monitoring tools can also be deployed, and the computing system can monitor the status of each computing resource in real time through the resource monitoring tools to ensure the smooth operation of parallel processing.

[0161] To demonstrate the improvement of video memory usage by the feature extraction method of sequence data in this specification, please refer to the theoretical calculation of video memory usage in the following example:

[0162] Set the dimension of the target sequence data (D) = 128, the number of attention heads (n_heads) = 1, each position stores a single-precision floating point number size of (float) = 4 bytes, and each expert network processes the Top-256 elements.

[0163] Specifically calculate and compare the theoretical video memory consumption of the self-attention network and the feature extraction method of sequence data in this specification when processing sequence data with a length of L=3000.

[0164] When using a self-attention network:

[0165] · Video memory usage ≈ L x L x D x sizeof(float)

[0166] ≈3000x 3000x 128x 4bytes

[0167] ≈3.456GB

[0168] When the feature extraction method for sequence data of this specification is used, the dimension (D) of the target sequence data and the number of attention heads (n_heads) are the same as the values ​​in the previous text, and each expert network processes 256 elements (i.e., K=256 in Top-K). Then, the video memory consumption of the feature extraction method for sequence data of this specification can be obtained by the following formula:

[0169] ·Memory usage per expert ≈ K x K x D x size of (float)

[0170] ·Video memory consumption per expert≈256x 256x 128x 4bytes

[0171] ·Video memory consumption per expert≈8.388608MB

[0172] Total video memory usage of 12 experts ≈ 12x 8.388608MB

[0173] The total memory usage of 12 experts is ≈100.663296MB (approximately 0.1GB)

[0174] Considering that when the input sequence length increases, the size of the attention score matrix grows quadratically, resulting in a rapid increase in video memory requirements. Therefore, the storage of the attention score matrix is ​​a major bottleneck for video memory usage, so when calculating video memory, we mainly consider the impact of the storage of the attention score matrix on video memory usage. If additional gated output calculations and accumulation processing are considered, the final result will increase slightly, but it is still far less than the consumption of the self-attention mechanism. Therefore, it is not difficult to see that the video memory usage of this manual is 1 / 35 of the original, so the feature extraction method of sequence data in this manual can greatly improve the depth and breadth of the model and improve the model effect.

[0175] Through the embodiments in the above specification, it can be seen that since each expert network only processes Top-K elements, the demand for video memory is significantly reduced, especially when processing large-scale data, the improvement of video memory occupancy will be more obvious; and for each expert network, the selection of Top-K reduces the computational complexity and improves the computational efficiency of the model on long sequences; each expert network can process the most relevant elements, so for each expert network, the model can better characterize the diversity of data, thereby improving the overall performance; and for the computing system, the number of expert networks and the Top-K value can be dynamically adjusted to adapt to different data and task requirements. The method provided in the embodiment of this specification greatly reduces the computational cost without sacrificing accuracy, and is particularly suitable for processing the long sequence computing challenges faced by large models, and realizing efficient resource utilization. Therefore, through the embodiments in this specification, not only the representation capability is retained, but also the computational complexity and video memory occupancy are reduced, not only the performance of the model is improved, but also the adaptability and scalability of the model in different scenarios are enhanced, not only the information loss problem that the self-attention mechanism variant may face is overcome, but also an efficient and scalable feature extraction method suitable for application scenarios of large models and long sequence data is provided.

[0176] On the other hand, this specification provides a computer-readable non-transitory storage medium, storing at least one instruction set for feature extraction of sequence data. When the at least one instruction set is executed by the processor, the at least one instruction set instructs the processor to implement the steps of the method P300 for feature extraction of sequence data described in this specification. In some possible embodiments, various aspects of this specification can also be implemented in the form of a program product, which includes program code. When the program product is run on a computing system, the program code is used to make the computing system perform the steps of the method P300 for feature extraction of sequence data described in this specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on a computing system. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in baseband as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operations of this specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may execute entirely on the computing system, partly on the computing system, as a stand-alone software package, partly on the computing system and partly on a remote computing device, or entirely on the remote computing device.

[0177] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0178] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0179] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0180] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. The content of the secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0181] Patents, patent applications, publications of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except those that are inconsistent or conflicting with this document or have a limiting effect on the broadest scope of the claims, are hereby incorporated by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0182] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for extracting features from sequence data, applied to a computing system in which multiple candidate expert networks are deployed, wherein different candidate expert networks are deployed on different computing resources in the computing system, the method comprising: Obtaining target sequence data, wherein the target sequence data includes a plurality of elements; Determine a plurality of subsequences corresponding to a plurality of target expert networks in the target sequence data, wherein the plurality of target expert networks are at least part of the plurality of candidate expert networks, each subsequence includes a portion of elements in the target sequence data, and the correlation between the elements in each subsequence and the corresponding target expert network satisfies the target condition, and the correlation represents the degree of suitability of the expert network for processing the elements; Processing the multiple subsequences using computing resources corresponding to the multiple target expert networks to obtain feature sub-results corresponding to each of the multiple subsequences; and Based on the characteristic sub-results corresponding to each of the multiple sub-sequences, a characteristic representation corresponding to the target sequence data is determined.

2. The method according to claim 1, wherein: Each element in the target sequence data corresponds to an element representation vector, the multiple target expert networks respectively correspond to expert representation vectors, and the multiple subsequences corresponding to the multiple target expert networks in the target sequence data are determined one by one, including: Determining the correlation between each target expert network in the multiple target expert networks and each element in the target sequence data based on the expert representation vectors of each target expert network and the element representation vector corresponding to each element in the target sequence data; and Based on the correlations between the multiple target expert networks and the elements in the target sequence data, the multiple subsequences corresponding to the multiple target expert networks are determined in the target sequence data.

3. The method according to claim 2, wherein: The determining, based on the correlation between the multiple target expert networks and each element in the target sequence data, the multiple subsequences corresponding to the multiple target expert networks in the target sequence data comprises: For each target expert network in the multiple target expert networks, multiple target elements are determined based on the correlation between the target expert network and each element in the target sequence data, and a subsequence corresponding to the target expert network is generated based on the multiple target elements, wherein: The multiple target elements are the first K elements in the target sequence data with the highest correlation with the target expert network, or the elements in the target sequence data with the correlation with the target expert network in a specified interval.

4. The method according to claim 2, wherein: The determining, based on the correlation between the multiple target expert networks and each element in the target sequence data, the multiple subsequences corresponding to the multiple target expert networks in the target sequence data comprises: For each element in the target sequence data, based on the correlation between the element and the multiple target expert networks, determining at least one target expert network suitable for processing the element from among the multiple target expert networks; and Based on the correlation between each element in the target sequence data and the at least one target expert network corresponding thereto, the multiple subsequences corresponding one-to-one to the multiple target expert networks are determined in the target sequence data.

5. The method according to claim 4, wherein: The step of determining the multiple subsequences corresponding to the multiple target expert networks one by one based on the correlation between each element in the target sequence data and the at least one target expert network corresponding thereto comprises: For each target expert network among the multiple target expert networks, based on the correlation between the target expert network and the element suitable for processing, multiple target elements are determined from the elements suitable for processing by the target expert network, and a subsequence corresponding to the target expert network is generated based on the multiple target elements, wherein: The multiple target elements are multiple elements in the target sequence data that have the highest correlation with the target expert network, or multiple elements in the target sequence data that have a correlation with the target expert network in a specified interval.

6. The method according to claim 4, wherein: At least one target expert network adapted to process said element comprises: the top N expert networks with the highest correlation with the element among the multiple target expert networks, where N is an integer greater than or equal to 1, or An expert network among the multiple target expert networks whose correlation with the element is greater than or equal to a preset threshold.

7. The method according to claim 1, wherein: The multiple target expert networks are some expert networks among the multiple candidate expert networks, and the method further includes: Determining the relevance between the target sequence data and the plurality of candidate expert networks; and Based on the correlation between the target sequence data and the multiple candidate expert networks, multiple candidate expert networks with the highest correlation with the target sequence data among the multiple candidate expert networks are determined as the multiple target expert networks.

8. The method according to any one of claims 3 or 5, wherein: After determining the multiple subsequences corresponding one-to-one to the multiple target expert networks, if there are elements to be processed in the target sequence data that are not included in any subsequence, the method further includes: adding the element to be processed to the subsequence with the least number of elements among the multiple subsequences; or, Creating a new target expert network in the computing system, and generating a subsequence corresponding to the new target expert network based on the elements to be processed; or, A new target expert network is selected from the multiple candidate expert networks, and a subsequence corresponding to the new target expert network is generated based on the element to be processed.

9. The method according to claim 1, wherein: The feature sub-result includes a feature vector corresponding to an element in the target sequence data, and determining the feature representation corresponding to the target sequence data based on the feature sub-results corresponding to each of the plurality of sub-sequences includes: For each element in the target sequence data, the relevance between the element and the multiple target expert networks is used as a weight, and the feature vectors corresponding to the element in each feature sub-result are weighted summed to determine the feature result of the element; Based on the feature results of each element in the target sequence data, a feature representation corresponding to the target sequence data is determined.

10. The method according to claim 1, wherein: The processing of the multiple subsequences by using computing resources corresponding to the multiple target expert networks includes: The multiple subsequences are simultaneously input into the respective corresponding target expert networks, so as to perform parallel processing on the multiple subsequences through the computing resources corresponding to the multiple target expert networks.

11. A feature extraction system for sequence data, comprising: At least one storage medium storing at least one instruction set for performing feature extraction; as well as At least one processor is communicatively connected to the at least one storage medium, wherein when the feature extraction system is running, the at least one processor reads the at least one instruction set and executes the feature extraction method as described in any one of claims 1-10 according to the instructions of the at least one instruction set.

12. A computer-readable non-volatile storage medium, wherein: The computer-readable non-volatile storage medium stores at least one instruction set, and when the at least one instruction set is executed by at least one processor, the feature extraction method for sequence data according to any one of claims 1 to 10 is implemented.

Images