Method, apparatus, device, and medium for determining similarity in text processing tasks
By employing a neural network with importance coefficients to scale network module outputs, the method addresses high computational costs in text processing task similarity, enabling efficient and accurate task similarity determination and related applications.
Patent Information
- Application Number
- JP2024135540
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-05-28
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-15
AI Technical Summary
Conventional methods for obtaining task embeddings of text processing tasks require high computational costs and memory demands, and fail to accurately evaluate similarity between different tasks.
A neural network is configured with multiple importance coefficients to scale the outputs of network modules, trained using text samples from text processing tasks, allowing for the determination of embedding features with low computational cost and storage overhead, and enabling more accurate task similarity measurements.
This approach allows for efficient and accurate determination of text processing task similarity with reduced computational and storage requirements, facilitating tasks such as task migration, clustering, and recommendation.
Smart Images

Figure 2024161486000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to technical fields such as natural language processing and deep learning, in particular to a method for determining text processing task similarity, an apparatus for determining text processing task similarity, an electronic device, a computer-readable storage medium and a computer program product. [Background technology]
[0002] Artificial intelligence is a discipline that allows computers to simulate certain human thought processes and intelligent behaviors (such as learning, training neural network models, thinking, planning, etc.), and includes both hardware and software level technologies. Artificial intelligence hardware technology generally includes sensors, artificial intelligence dedicated chips, cloud computing, distributed storage, big data processing, and other technologies, while artificial intelligence software technology mainly includes natural language processing technology, computer vision technology, voice recognition technology, and several directions such as machine learning / deep learning, big data processing technology, and knowledge graph technology.
[0003] The methods described in this section are not necessarily methods that have been previously conceived or adopted. Unless otherwise noted, any methods described in this section should not be considered prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise noted, any problem addressed in this section should not be considered an admission of prior art. Summary of the Invention
[0004] The present disclosure provides a method for determining text processing task similarity, an apparatus for determining text processing task similarity, an electronic device, a computer-readable storage medium and a computer program product.
[0005] According to one aspect of the present disclosure, a method for determining a text processing task similarity is provided, the method including: determining a first task, a second task, and a neural network to be trained, the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, the plurality of importance coefficients being used to scale an output value of the corresponding network module, respectively; performing a target operation with the first task and the second task as target tasks, respectively, to obtain embedding features of the first task and the second task, the target operation including: training the neural network to be trained using text samples corresponding to the target task, and obtaining a plurality of trained importance coefficients; determining an embedding feature of the target task based on the trained plurality of importance coefficients; and determining a task similarity between the first task and the second task based on the embedding features of the first task and the second task, respectively.
[0006] According to another aspect of the present disclosure, an apparatus for determining a text processing task similarity is provided, the apparatus including: a first determination unit configured to determine a first task, a second task, and a neural network to be trained, the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, the plurality of importance coefficients being respectively used to scale an output value of a corresponding network module; an embedding feature acquisition unit including: a training subunit configured to perform a target operation with the first task and the second task as target tasks, respectively, to obtain embedding features for the first task and the second task, the training subunit configured to train the neural network to be trained using a text sample corresponding to the target task and obtain the trained plurality of importance coefficients; and a first determination subunit configured to determine an embedding feature for the target task based on the trained plurality of importance coefficients; and a second determination unit configured to determine a task similarity between the first task and the second task based on the embedding features for the first task and the second task, respectively.
[0007] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, where the memory stores instructions executable by the at least one processor, which instructions, when executed by the at least one processor, enable the at least one processor to perform the method described above.
[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method described above.
[0009] According to another aspect of the present disclosure there is provided a computer program product comprising a computer program which, when executed by a processor, implements the above-mentioned method.
[0010] According to one or more embodiments of the present disclosure, the present disclosure realizes obtaining embedding features of different text processing tasks with low computational cost and storage overhead by setting multiple importance coefficients in a neural network to scale the outputs of multiple network modules, and using text samples of text processing tasks to train the neural network, and then determining embedding features of the text processing tasks based on the importance coefficients after training. Also, by training neural networks with the same structure using text samples of different text processing tasks, the embedding features of the tasks can be obtained, and more accurate task similarity can be obtained.
[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. Other features of the present disclosure will be easily understood from the following description. [Brief description of the drawings]
[0012] The drawings illustratively illustrate the examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and are not intended to limit the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1] 1 shows a schematic diagram of an example system capable of implementing various methods described herein, according to embodiments of the present disclosure. [Diagram 2] 2 shows a flowchart of a method for determining text processing task similarity according to an exemplary embodiment of the present disclosure. [Diagram 3] 2 shows a flowchart of a method for determining text processing task similarity according to an exemplary embodiment of the present disclosure. [Figure 4]4 shows a flowchart for training a neural network to be trained according to an exemplary embodiment of the present invention. [Diagram 5] 4 shows a flowchart for training a neural network to be trained according to an exemplary embodiment of the present invention. [Figure 6] 1 shows a structural block diagram of an apparatus for determining text processing task similarity according to an exemplary embodiment of the present invention; [Figure 7] FIG. 1 illustrates a structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, and various details in the embodiments of the present disclosure contained therein are to be considered as merely illustrative, in order to facilitate understanding. Therefore, it should be recognized that those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description omits descriptions of known functions and structures.
[0014] In this disclosure, unless otherwise stated, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of an element, or in some cases, may refer to different instances based on the contextual description.
[0015] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless specifically limited in number. It should be noted that the term "and / or" as used in the present disclosure covers any and all possible combinations of the listed items.
[0016] A text processing task (hereafter simply called a task) refers to a specific task that requires processing and analyzing text data. By obtaining the embedding features of a task (a task-specific vector representation), it is possible to measure the similarity between different text processing tasks.
[0017] In the related art, the conventional methods for obtaining task embeddings of text processing tasks all require high computational costs and memory demands, and the accuracy of evaluating the similarity between different tasks needs to be improved.
[0018] To solve the above problem, a neural network is set with multiple importance coefficients for scaling the outputs of multiple network modules, and the neural network is trained using text samples of text processing tasks, and the embedding features of the text processing tasks are determined based on the importance coefficients after training, thereby achieving the acquisition of embedding features of different text processing tasks with low computational cost and memory overhead. Furthermore, the same structure of neural networks can be trained using text samples of different text processing tasks to obtain the embedding features of the tasks, and more accurate task similarity can be obtained.
[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
[0020] 1 illustrates a schematic diagram of an example system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. With reference to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
[0021] In an embodiment of the present disclosure, server 120 may execute one or more services or software applications that enable the performance of the methods of the present disclosure.
[0022] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.
[0023] In the arrangement shown in FIG. 1, server 120 may include one or more assemblies that implement the functions performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed on one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn use one or more client applications to interact with server 120 to utilize the services provided by these assemblies. It should be understood that a variety of different system arrangements are possible and may differ from system 100. Thus, FIG. 1 is an example of a system for implementing various methods described herein and is not intended to be limiting.
[0024] A user may use client devices 101, 102, 103, 104, 105, and / or 106 to conduct human-machine interactions. A client device may provide an interface through which a user of the client device interacts with the client device. The client device may also output information to the user through the interface. Although only six client devices are illustrated in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.
[0025] The client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., GOOGLE Chrome OS), and may include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android, and the like. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.
[0026] Network 110 may be any type of network known to those skilled in the art that may use any one of a number of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.
[0027] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframes, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage devices). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.
[0028] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercially available server operating system. Server 120 may also run any one of a variety of additional server applications and / or mid-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.
[0029] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.
[0030] In some embodiments, the server 120 may be a server of a distributed system or a server incorporating blockchain. The server 120 may be a cloud server, or an intelligent cloud computing server or an intelligent cloud host equipped with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, and solves the defects of the traditional physical host and virtual private server (VPS) services, such as high management difficulty and low business scalability.
[0031] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of a variety of types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the database and from the database in response to instructions.
[0032] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be various types of databases, such as a key-value repository, an object repository, a generic repository supported by a file system, etc.
[0033] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the application of the various methods and apparatus described in accordance with this disclosure.
[0034] According to one aspect of the present disclosure, a method for determining text processing task similarity is provided. FIG. 2 shows a flowchart of a method 200 for determining text processing task similarity according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the method includes steps S201 to S203. In S201, a first task, a second task, and a neural network to be trained are determined, where the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, and the plurality of importance coefficients are respectively used to scale the output values of the corresponding network modules. In S202, a target operation is performed using the first task and the second task as target tasks to obtain respective embedding features of the first task and the second task, the target operation includes a step S2021 of training the neural network to be trained using text samples corresponding to the target task and obtaining the trained plurality of importance coefficients, and a step S2022 of determining an embedding feature of the target task based on the trained plurality of importance coefficients. In S203, a task similarity between the first task and the second task is determined based on the respective embedding features of the first task and the second task. It should be understood that steps S2021 and S2022 may be substeps of step S202.
[0035] Thus, by setting multiple importance coefficients in the neural network to scale the outputs of multiple network modules, and using text samples of text processing tasks to train the neural network, and then determining the embedding features of the text processing tasks based on the importance coefficients after training, it is possible to obtain the embedding features of different text processing tasks with low computational cost and storage overhead. Also, by using text samples of different text processing tasks to train neural networks with the same structure, the embedding features of the tasks can be obtained, and more accurate task similarity can be obtained.
[0036] As mentioned above, a text processing task refers to a specific task that requires processing and analyzing text data, and can include, for example, text classification, sentiment analysis, named entity recognition, summary generation, etc. These text processing tasks can be represented by task embeddings (task-specific vector representations). The task embeddings of different tasks can construct a semantic space to realize a measure of similarity between different text processing tasks.
[0037] According to some embodiments, after determining the task similarity between the first task and the second task in step S203, the two tasks can be downstream processed based on the similarity between the two tasks. As shown in Fig. 3, the method can further include step S304 of performing task transition between the first task and the second task in response to determining that the task similarity between the first task and the second task is greater than a preset similarity. Note that, for the operations and effects of steps S301 to S303 and their sub-steps in Fig. 3, the description of steps S201 to S203 and their sub-steps in Fig. 2 can be referred to, and the description will be omitted here.
[0038] Task migration refers to utilizing knowledge or models from one task to improve the performance of another related task. For example, models and data from an existing news classification task can be used to assist in a new news classification task, such as migrating from a sports news classification task to an entertainment news classification task, or an existing movie review sentiment analysis model can be used to improve the effectiveness of product review sentiment analysis. The above method allows the correlation between tasks to be obtained quickly, automatically, and accurately, thereby achieving efficient task migration.
[0039] In some embodiments, task transfer can be achieved through data augmentation, i.e., the training set of a source task (first task) is used to augment the training set of a target task (second task), thereby improving the generalization ability of the model by increasing the data of related tasks, especially when the target task data is sparse.
[0040] In some embodiments, task transfer can be achieved through a model parameter transfer method, i.e., some or all of the model parameters trained in a source task (first task) are transferred to a model of a target task (second task), thereby reducing the training time and resources of the target task.
[0041] In some embodiments, task transfer can be achieved through joint training; That is, models for the source task (first task) and the target task (second task) are trained simultaneously and some network structures or parameters are shared to improve the performance of the two tasks.
[0042] As can be seen, a reasonable source task can significantly improve the performance of a target task, and similarly, a wrong source task can also impair the performance of a target task. Therefore, the selection of source tasks is important in task migration. Using the method disclosed herein, a large number of source tasks with high similarity to a target task can be quickly found with low computational cost and memory overhead.
[0043] In some embodiments, after determining the text processing task similarity, task clustering and task recommendation can be further performed.
[0044] Task clustering is, after determining task similarity between a plurality of tasks including a first task and a second task, clustering the plurality of tasks based on the task similarity between the plurality of tasks, so that similar tasks are grouped together, which is helpful for organizing and managing a large number of tasks, especially in the scenario of multi-task learning or multi-task management.
[0045] In some platforms (e.g., machine learning model sharing platforms), task embeddings can be used to recommend related tasks or models. In one exemplary embodiment, based on the embedding features of a first task that a user is processing, task similarity is determined for multiple other tasks as a second task, thereby recommending tasks and models similar to the first task, and helping users quickly find related resources (datasets, models, etc. of similar tasks).
[0046] It should be understood that the method for determining similarity in text processing tasks provided by the present disclosure can also be used in richer scenes and is not limited thereto.
[0047] In step S201, the first task and the second task may be two pre-selected text processing tasks whose task similarity needs to be determined. The neural network model to be trained may be any neural network model available for text processing.
[0048] According to some embodiments, the neural network to be trained is a Transformer architecture, and the plurality of network modules includes a plurality of self-attention modules and a plurality of feedforward neural network modules.
[0049] In the neural network of the Transformer architecture, each layer mainly contains two parts: a multi-head attention module and a feed forward neural network module. The multi-head attention mechanism can be formalized as follows:
number
[0050] A feedforward neural network module can be formalized as follows:
number
[0051] A learnable importance coefficient m for each self-attention head H Each feedforward neural network can be assigned a learnable importance coefficient m F Assign.
number
[0052] After learning, the above-mentioned importance coefficient can scale the output value of the corresponding network module, thereby realizing the retention or expansion of the output value of the important network module, and the reduction of the output value of the network module with less impact on the output.
[0053] In step S202, the first task and the second task are respectively taken as target tasks, and a target operation is performed to obtain the embedding features of the first task and the second task. As described above, the target operation can include steps S2021 and S2022.
[0054] Before performing step S2021, a training set or data set corresponding to the target task may be obtained and may include a number of text samples. The text samples may include text data and may include corresponding ground truth tags.
[0055] In step S2021, the text data in the text sample is input to the neural network to be trained, and the text processing result output by the neural network to be trained can be obtained. Furthermore, the parameters of the neural network to be trained can be adjusted according to the difference between the text processing result and the real tab, including the learnable parameters and the importance coefficients of the multiple network modules. After the training of the neural network to be trained is completed, the trained importance coefficients can be obtained.
[0056] In step S2022, the trained importance coefficients can be directly determined as the embedding features of the target task.
[0057] According to some embodiments, the neural network to be trained is a large language model. For a large language model, the computational and storage costs of obtaining task embeddings by conventional methods are very high, but with the method of the present disclosure, the amount of data that needs to be stored is only the number of multi-head attention modules and feedforward neural networks in the large language model. In some embodiments, the importance coefficient can be a Boolean variable (described below) to further reduce the amount of data that needs to be stored.
[0058] According to some embodiments, the initial values of the importance coefficients are randomly initialized, and the neural network to be trained is trained using text samples of different tasks, so that the randomly initialized importance coefficients converge to a combination of importance coefficients corresponding to different tasks.
[0059] In some embodiments, as shown in FIG. 4, in step S2021, training a neural network to be trained using text samples corresponding to a target task and obtaining a plurality of trained importance coefficients includes steps S401 to S403. In step S401, a first loss value is determined based on the plurality of importance coefficients, and the first loss value has a positive correlation with the absolute value of the plurality of importance coefficients. In step S402, a text processing result output by the neural network to be trained based on the text samples is obtained, and a second loss value is determined based on the text processing result, and the second loss value is used to evaluate the text processing result. In step S403, the plurality of importance coefficients and learnable parameters of a plurality of network modules are adjusted based on the first loss value and the second loss value.
[0060] Considering that neural networks are sparse, i.e., the network modules that play important roles in neural networks are sparse, the trained multiple importance coefficients should also be sparse. To ensure the sparsity of the importance coefficients, one regularization term, i.e., the first loss value, can be added to the training objective.
number
number
number
[0061] In some embodiments, as shown in FIG. 5, in step S2021, training the neural network to be trained using text samples corresponding to the target task and obtaining a plurality of trained importance coefficients may include steps S504 to S505. In step S504, after performing parameter adjustment for a predetermined number of iterations, converting the current plurality of importance coefficients into a plurality of indication values based on a predetermined threshold value. In step S505, in response to determining that the changes in the plurality of indication values satisfy a predetermined rule after the parameter adjustment for a plurality of consecutive predetermined number of iterations, training of the neural network to be trained is stopped. Note that the operations of steps S501 to S503 in FIG. 5 may refer to the above description of steps S401 to S403, and will not be described here.
[0062] In some embodiments, the importance coefficient can be converted into an indication value, These indicators can converge early (not more than one round, i.e., not more than one traversal of the complete training set). Therefore, an early stopping strategy can be adopted to stop the model at an early training point according to the above rules. In this case, the neural network itself may not have yet converged, i.e., the neural network has not yet fully completed the corresponding target task, but the indicators obtained by converting the importance coefficients have already converged.
[0063] The preset number of iterations is sometimes called a mini-epoch or micro-epoch, and its value can be set as needed and is not limited here. In step S505, after parameter adjustment for a plurality of consecutive N preset iterations, in response to determining that the changes in the plurality of indication values satisfy the preset rule, the training of the neural network to be trained is stopped. Note that the value of N can be set as needed and is not limited here.
[0064] In some embodiments, the indicator value may take a value of 0 or 1, and an importance factor that exceeds a preset threshold may be converted to 1, and an importance factor that does not exceed the preset threshold may be converted to 0.
[0065] In some embodiments, the plurality of instruction values are all Boolean variables, and the predetermined rule includes that after a plurality of consecutive predetermined iterations of parameter adjustment, a number of changing instruction values among the plurality of instruction values does not exceed the predetermined parameter.
[0066] In some embodiments, the training can be stopped when the change in the mask for several consecutive mini-epochs does not exceed a fixed parameter γ. It can be understood that the value of γ can be set as needed and is not limited here.
[0067] In some embodiments, in step S2022, determining the embedding features of the target task based on the trained importance coefficients may include determining the instruction values obtained in the final transformation as the embedding features of the target task.
[0068] In some embodiments, the indication value obtained from the final transformation of the importance coefficients corresponding to the multiple self-attention mechanism modules and the indication value obtained from the final transformation of the importance coefficients corresponding to the multiple feedforward neural network modules can be concatenated to obtain an embedding feature of the target task.
[0069] In some embodiments, in step S203, determining the task similarity between the first task and the second task based on the embedded features of the first task and the second task, respectively, may include determining the task similarity between the first task and the second task based on the number of instruction values that have the same position and value in the embedded features of the first task and the embedded features of the second task.
[0070] The same positional instruction value means that it corresponds to the same network module (self-attention head or feedforward neural network), and the same value means that the network module is not important in either the first task or the second task. Therefore, the above method can easily and quickly obtain accurate task similarity.
[0071] According to another aspect of the present disclosure, an apparatus for determining a text processing task similarity is provided. As shown in Fig. 6, the apparatus 600 includes: a first determining unit 610 configured to determine a first task, a second task, and a neural network to be trained, the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, and the plurality of importance coefficients are respectively used to scale the output value of the corresponding network module; a training subunit 622 configured to perform a target operation with the first task and the second task as a target task to obtain respective embedding features of the first task and the second task, and to train the neural network to be trained using text samples corresponding to the target task and obtain the trained plurality of importance coefficients; a first determining subunit 624 configured to determine an embedding feature of the target task based on the trained plurality of importance coefficients; and a second determining unit 630 configured to determine a task similarity between the first task and the second task based on the respective embedding features of the first task and the second task.
[0072] It can be understood that the operations and effects of units 610 to 630 and their sub-units in the apparatus 600 can refer to the above description of steps S201 to S203 and their sub-steps, and will not be described here.
[0073] According to some embodiments, the apparatus for determining text processing task similarity may further include a task transition unit configured to perform task transition between the first task and the second task in response to determining that the task similarity between the first task and the second task is greater than a predetermined similarity.
[0074] According to some embodiments, task transfer includes at least one of: using a training set for the first task to enhance a training set for the second task; transferring at least a portion of model parameters in a trained neural network for the first task to a neural network for the second task; and simultaneously training the neural network for the first task and the neural network for the second task, where the neural network for the first task and the neural network for the second task share some structure or parameters.
[0075] In some embodiments, the neural network to be trained is a Transformer architecture, and the multiple network modules may include multiple self-attention modules and multiple feedforward neural network modules.
[0076] In some embodiments, the neural network to be trained may be a large language model.
[0077] In some embodiments, the initial values of the importance coefficients may be obtained by random initialization.
[0078] In some embodiments, the training subunit may include: a second determination subunit configured to determine a first loss value based on a plurality of importance coefficients, the first loss value being positively correlated with an absolute value of the plurality of importance coefficients; an acquisition subunit configured to acquire a text processing result output by the neural network to be trained based on a text sample, and to determine a second loss value based on the text processing result, the second loss value being used to evaluate the text processing result; and a parameter adjustment subunit configured to adjust the plurality of importance coefficients and learnable parameters of the plurality of network modules based on the first loss value and the second loss value.
[0079] In some embodiments, the training subunit may include a conversion subunit configured to convert the current plurality of importance coefficients into a plurality of indication values based on a predetermined threshold value after a predetermined number of iterations of parameter adjustment, and an early stopping subunit configured to stop training of the neural network to be trained in response to changes in the plurality of indication values satisfying a predetermined rule after a plurality of successive predetermined number of iterations of parameter adjustment.
[0080] In some examples, the plurality of instruction values are all Boolean variables, and the preset rule may include that after a plurality of consecutive preset iterations of parameter adjustment, a number of changing instruction values among the plurality of instruction values does not exceed the preset parameter.
[0081] According to some embodiments, the first determination subunit may include a third determination subunit configured to determine the plurality of instruction values obtained in the final transformation as embedded features of the target task.
[0082] According to some embodiments, the second determination unit may include a fourth determination subunit configured to determine task similarity between the first task and the second task based on the number of instruction values that have the same position and the same value in the embedded feature of the first task and the embedded feature of the second task.
[0083] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.
[0084] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided.
[0085] Next, referring to FIG. 7, a block diagram of an electronic device 700 functioning as a server or client of the present disclosure will be described, which is an example of a hardware device applicable to each aspect of the present disclosure. The electronic device may represent various forms of digital electronic computers, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may further represent various forms of mobile devices, such as personal digital processing, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.
[0086] 7, the electronic device 700 includes a computing unit 701 that can perform various appropriate operations and processes according to a computer program stored in a read only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0087] A number of components in the electronic device 700, including an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709, are connected to the I / O interface 705. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input numeric or character information and generate key signal inputs for user settings and / or function control of the electronic device, including but not limited to a mouse, a keyboard, a touch screen, a track board, a track ball, a control lever, a microphone, and / or a remote control. The output unit 707 may be any type of device capable of presenting information, including but not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a magnetic disk, an optical disk. The communications unit 709 enables the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communications device, a wireless communications transceiver and / or chipset, such as, for example, a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communications device, and / or the like.
[0088] The computing unit 701 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes each of the methods, processes, and / or processes described above. For example, in some embodiments, these methods, processes, and / or processes can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the methods, processes, and / or processes described above can be executed. Alternatively, in other embodiments, computing unit 701 is configured to perform these methods, processes, and / or operations in any other suitable manner (eg, by firmware).
[0089] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, and at least one output device, and may transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0090] The program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations specified in the flow charts and / or block diagrams. The program code may be entirely executed on a machine, partly executed on a machine, partly executed on a machine as a separate software package and partly executed on a remote machine, or entirely executed on a remote machine or server.
[0091] In the context of the present disclosure, a machine-readable medium may be a tangible medium, which may include or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0092] To provide for interaction with a user, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback) and may receive input from the user in any form (including sound input, speech input, or tactile input).
[0093] The systems and techniques described herein may be implemented in a computing system including a backstage component (e.g., as a data server), a middleware component (e.g., an application server), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique embodiment), or any combination of the backstage component, middleware component, or front-end component. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0094] The computer system may include a client side and a server. The client side and the server are generally remote from each other and usually interact with each other via a communication network. The relationship between the client side and the server is generated by running computer programs having a client side-server relationship with each other on corresponding computers. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0095] It should be understood that the steps may be reordered, added, or removed using the various forms of flow described above. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, and the present disclosure is not limited thereto as long as the technical solutions disclosed in this disclosure can achieve the desired results.
[0096] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents after grant. Various elements of the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.
Claims
1. 1. A method for determining text processing task similarity, comprising: determining a first task, a second task, and a neural network to be trained, the neural network to be trained including a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, the plurality of importance coefficients being used to scale output values of the respective corresponding network modules; and performing a target operation with the first task and the second task as target tasks, respectively, to obtain embedding features of the first task and the second task, the target operation including: Training the neural network to be trained using text samples corresponding to the target task, and obtaining a number of trained importance coefficients; determining an embedding feature of the target task based on the trained importance coefficients; and determining a task similarity between the first task and the second task based on respective embedding features of the first task and the second task.
2. Training the neural network to be trained using text samples corresponding to the target task and obtaining a plurality of trained importance coefficients, determining a first loss value based on the plurality of importance coefficients, the first loss value being positively correlated with an absolute value of the plurality of importance coefficients; Obtain a text processing result output by the neural network to be trained according to the text sample, and determine a second loss value according to the text processing result, where the second loss value is used to evaluate the text processing result; and adjusting the plurality of importance coefficients and learnable parameters of the plurality of network modules based on the first loss value and the second loss value.
3. Training the neural network to be trained using text samples corresponding to the target task and obtaining a plurality of trained importance coefficients, After performing the parameter adjustment for a preset number of iterations, converting the current plurality of importance coefficients into a plurality of indication values based on a preset threshold value; and ceasing training of the neural network to be trained in response to determining that changes in a plurality of indicator values satisfy a predetermined rule after a plurality of successive parameter adjustments for the predetermined number of iterations.
4. 4. The method of claim 3, wherein the plurality of instruction values are Boolean variables, and the predetermined rule includes that after a plurality of successive parameter adjustments for the predetermined number of iterations, a number of the instruction values that change among the plurality of instruction values does not exceed a predetermined parameter.
5. determining an embedding feature for the target task based on the trained importance coefficients; The method of claim 3 , further comprising determining a plurality of instruction values obtained in the final transformation as embedded features of the target task.
6. Determining task similarity of the first task and the second task based on respective embedding features of the first task and the second task includes:
6. The method of claim 5, further comprising determining task similarity between the first task and the second task based on a number of instruction values that are the same in position and value in the embedded features of the first task and the embedded features of the second task.
7. The method of claim 1 , wherein the neural network to be trained is a Transformer architecture, and the plurality of network modules includes a plurality of self-attention modules and a plurality of feedforward neural network modules.
8. The method of claim 7 , wherein the neural network to be trained is a large language model.
9. The method of claim 1 , wherein the initial values of the plurality of importance coefficients are randomly initialized.
10. 2. The method of claim 1, further comprising: in response to determining that a task similarity between the first task and the second task is greater than a predetermined similarity, performing a task transition between the first task and the second task.
11. The task transition includes: augmenting a training set for the second task using a training set for the first task; transferring at least a portion of model parameters in the trained neural network for the first task to a neural network for the second task; and simultaneously training the neural network for the first task and the neural network for the second task, wherein the neural network for the first task and the neural network for the second task share some structure or parameters.
12. An apparatus for determining a text processing task similarity, comprising: a first determination unit configured to determine a first task, a second task, and a neural network to be trained, the neural network to be trained including a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, the plurality of importance coefficients being used to scale an output value of a respective corresponding network module; A target operation is performed with the first task and the second task as target tasks, respectively, to obtain embedding features of the first task and the second task, respectively; a training subunit configured to train the neural network to be trained using text samples corresponding to the target task and obtain a plurality of trained importance coefficients; an embedding feature acquisition unit including a first determination subunit configured to determine an embedding feature of the target task based on the trained plurality of importance coefficients; and a second determination unit configured to determine a task similarity between the first task and the second task based on respective embedding features of the first task and the second task.
13. The training subunit comprises: a second determination subunit configured to determine a first loss value based on the plurality of importance coefficients, the first loss value being positively correlated with the absolute value of the plurality of importance coefficients; an acquisition subunit configured to acquire a text processing result output by the neural network to be trained based on the text sample, and determine a second loss value based on the text processing result, the second loss value being used to evaluate the text processing result; and a parameter adjusting subunit configured to adjust the plurality of importance coefficients and learnable parameters of the plurality of network modules based on the first loss value and the second loss value.
14. The training subunit comprises: a conversion subunit configured to convert the current plurality of importance coefficients into a plurality of indication values according to a preset threshold value after performing parameter adjustment for a preset number of iterations; and an early stopping subunit configured to stop training of the neural network to be trained in response to determining that changes in a plurality of indicator values satisfy a predetermined rule after parameter adjustment for a plurality of successive said predetermined iterations.
15. 15. The apparatus of claim 14, wherein the plurality of instruction values are all Boolean variables, and the predetermined rule includes that after parameter adjustment for a plurality of successive predetermined iterations, a number of changing instruction values among the plurality of instruction values does not exceed a predetermined parameter.
16. The first determining subunit is The apparatus of claim 14 , further comprising a third determination subunit configured to determine a plurality of instruction values obtained in the final transformation as embedded features of the target task.
17. The second determination unit is 17. The apparatus of claim 16, further comprising a fourth determination subunit configured to determine task similarity between the first task and the second task based on a number of instruction values that have the same position and the same value in the embedded feature of the first task and the embedded feature of the second task.
18. The apparatus according to any one of claims 12 to 17, wherein the neural network to be trained is a Transformer architecture, and the plurality of network modules includes a plurality of self-attention modules and a plurality of feedforward neural network modules.
19. 20. The apparatus of claim 18, wherein the neural network to be trained is a large language model.
20. The apparatus according to any one of claims 12 to 17, wherein the initial values of the plurality of importance coefficients are obtained by random initialization.
21. 18. The apparatus of claim 12, further comprising a task migration unit configured to perform a task migration between the first task and the second task in response to determining that a task similarity between the first task and the second task is greater than a preset similarity.
22. The task transition includes: augmenting a training set for the second task using a training set for the first task; transferring at least a portion of model parameters in the trained neural network for the first task to a neural network for the second task; and simultaneously training the neural network for the first task and the neural network for the second task, wherein the neural network for the first task and the neural network for the second task share some structure or parameters.
23. An electronic device, comprising: At least one processor; a memory communicatively coupled to the at least one processor; An electronic device in which instructions executable by the at least one processor are stored in the memory, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.
24. A non-transitory computer readable storage medium having stored thereon computer instructions, the computer instructions causing a computer to perform the method of any one of claims 1 to 11.
25. A computer program product comprising a computer program which, when executed by a processor, performs the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Comparative learning method and device of graph structure and computer storage medium
CN115129945A
Tree-based transfer learning of tunable parameters
US20220180241A1
Frozen Model Adaptation Through Soft Prompt Transfer
US20240020546A1
