Methods, apparatus, devices, and media for determining similarity in text processing tasks

Abstract

To provide a method, an apparatus, a device, a storage medium, and a program for determining similarity in text processing tasks in natural language processing and deep learning.SOLUTION: A method comprises: determining a first task, a second task, and a neural network to be trained; performing a target operation with the first task and the second task as target tasks, respectively, to obtain embedding features of the first task and the second task; training a neural network to be trained using text samples corresponding to the target task, and obtaining a plurality of trained importance coefficients; determining an embedding feature of the target task based on the plurality of trained importance coefficients; and determining the task similarity between the first task and the second task based on the embedding features of the first task and the second task, respectively.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This disclosure relates to the field of artificial intelligence, and more particularly to the field of natural language processing and deep learning, and more specifically to methods for determining the similarity of text processing tasks, apparatus for determining the similarity of text processing tasks, electronic equipment, computer-readable storage media, and computer program products. [Background technology]

[0002] Artificial intelligence is the field of study that enables computers to simulate certain human thought processes and intelligent actions (e.g., learning, training neural network models, thinking, planning, etc.), and it encompasses both hardware-level and software-level technologies. Hardware technologies for artificial intelligence generally include technologies such as sensors, dedicated chips for artificial intelligence, cloud computing, distributed storage, and big data processing, while software technologies for artificial intelligence mainly include several areas such as natural language processing, computer vision, speech recognition, and machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] The methods described in this section are not necessarily previously conceived or adopted. Unless otherwise specified, none of the methods described in this section should be considered prior art simply because they are included in this section. Similarly, unless otherwise specified, none of the problems mentioned in this section should be considered accepted in the prior art. [Overview of the Initiative]

[0004] This disclosure provides a method for determining the similarity of text processing tasks, an apparatus for determining the similarity of text processing tasks, electronic equipment, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a method is provided for determining the similarity of text processing tasks, which includes determining a first task, a second task, and a neural network to be trained, wherein the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, the plurality of importance coefficients being used to scale the output values ​​of the respective network modules, and performing a target operation with the first task and the second task as target tasks to obtain the embedding features of the first task and the second task, respectively, the target operation including training the neural network to be trained using text samples corresponding to the target task and obtaining the trained plurality of importance coefficients, determining the embedding features of the target task based on the trained plurality of importance coefficients, and determining the task similarity of the first task and the second task based on the embedding features of the first task and the second task, respectively.

[0006] According to another aspect of the present disclosure, an apparatus for determining text processing task similarity is provided, comprising: a first determination unit configured to determine a first task, a second task, and a neural network to be trained, wherein the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, and the plurality of importance coefficients are used to scale the output values ​​of the respective network modules; an embedding feature acquisition unit configured to perform a target operation with the first task and the second task as target tasks to obtain the respective embedding features of the first task and the second task, train the neural network to be trained using text samples corresponding to the target tasks, and obtain the trained plurality of importance coefficients; a first determination subunit configured to determine the embedding features of the target task based on the trained plurality of importance coefficients; and a second determination unit configured to determine the task similarity between the first task and the second task based on the respective embedding features of the first task and the second task.

[0007] Another aspect of the present disclosure provides an electronic device comprising at least one processor and a memory communicated to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and these instructions are executed by the at least one processor, thereby enabling the at least one processor to perform the method described above.

[0008] Another aspect of this disclosure provides a non-temporary computer-readable storage medium storing computer instructions for causing a computer to perform the method described above.

[0009] According to another aspect of this disclosure, a computer program product is provided which includes a computer program, and the computer program, when executed by a processor, accomplishes the method described above.

[0010] According to one or more embodiments of this disclosure, the disclosure enables obtaining embedding features for different text processing tasks with low computational cost and memory overhead by setting multiple importance factors in a neural network to scale the outputs of multiple network modules, training the neural network using text samples of a text processing task, and further determining the embedding features of the text processing task based on the post-training importance factors. Furthermore, by training a neural network with the same structure using text samples from different text processing tasks, task embedding features can be obtained, resulting in more accurate task similarity.

[0011] It should be understood that the content described in this section is not intended to identify the essential or important features of the embodiments of this disclosure, nor is it intended to limit the scope of protection of this disclosure. Other features of this disclosure will be readily apparent from the following specification. [Brief explanation of the drawing]

[0012] The drawings illustrate embodiments and constitute part of the specification, and are used to illustrate exemplary embodiments of the embodiments together with the textual description of the specification. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to elements that are similar but not necessarily identical. [Figure 1] A schematic diagram of an exemplary system capable of carrying out the various methods described herein, according to the embodiments of this disclosure, is shown. [Figure 2] A flowchart illustrating a method for determining the similarity of a text processing task according to an exemplary embodiment of this disclosure is shown. [Figure 3] A flowchart illustrating a method for determining the similarity of a text processing task according to an exemplary embodiment of this disclosure is shown. [Figure 4]A flowchart for training a neural network to be trained, according to an exemplary embodiment of the present invention, is shown. [Figure 5] A flowchart for training a neural network to be trained, according to an exemplary embodiment of the present invention, is shown. [Figure 6] This shows a structural block diagram of an apparatus for determining the similarity of text processing tasks according to an exemplary embodiment of the present invention. [Figure 7] An exemplary structural block diagram of an electronic device that may be used to implement embodiments of the present disclosure is shown. [Modes for carrying out the invention]

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, and the various details of the embodiments of the present disclosure contained herein should be considered as illustrative only, for the purpose of aiding understanding. Accordingly, those skilled in the art should be aware that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of known functions and structures have been omitted in the following description.

[0014] In this disclosure, unless otherwise specified, terms such as “first,” “second,” etc., used to describe various elements are not intended to limit the spatial, timing, or importance relationships of these elements. Such terms are used solely to distinguish one element from another. In some examples, the first and second elements may refer to the same example of an element, or, in some cases, different examples based on the contextual description.

[0015] The terms used in the description of various examples of the present disclosure are for the sole purpose of describing specific examples and are not intended to be limiting. Unless otherwise clearly indicated in the context, elements may be one or plural, provided that the number of elements is not particularly limited. The term "and / or" used in the present disclosure covers any one of the listed items and all possible combinations.

[0016] A text processing task (hereinafter simply referred to as a task) refers to a specific operation that requires processing and analyzing text data. By obtaining the embedded features (task-specific vector representations) of a task, it is possible to measure the similarity between different text processing tasks.

[0017] In related technologies, all methods for obtaining the task embeddings of conventional text processing tasks require high computational costs and memory requirements, and there is a need to improve the evaluation accuracy of the similarity between different tasks.

[0018] To solve the above problems, a plurality of importance coefficients for scaling the outputs of a plurality of network modules in a neural network are set, and the neural network is trained using text samples of text processing tasks. Furthermore, by determining the embedded features of text processing tasks based on the importance coefficients after training, it is possible to obtain the embedded features of different text processing tasks with low computational costs and memory overheads. Also, by training a neural network with the same structure using text samples of different text processing tasks, the embedded features of tasks can be obtained, and a more accurate task similarity can be obtained.

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0020] FIG. 1 shows a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein can be implemented, according to an embodiment of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more applications.

[0021] In an embodiment of the present disclosure, the server 120 can execute one or more services or software applications that enable the execution of the methods of the present disclosure.

[0022] In some embodiments, the server 120 can also provide other services or software applications that can include a non-virtual environment and a virtual environment. In some embodiments, these services can be provided as web-based services or cloud services, for example, provided to users of the client devices 101, 102, 103, 104, 105, and / or 106 in a software as a service (SaaS) model.

[0023] In the arrangement shown in FIG. 1, the server 120 may include one or more assemblies that implement the functions executed by the server 120. These assemblies may include software assemblies, hardware assemblies, or combinations thereof that can be executed by one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can interact with the server 120 by sequentially using one or more client applications to utilize the services provided by these assemblies. It should be understood that various different system arrangements are possible and may differ from the system 100. Therefore, FIG. 1 is an example of a system for implementing the various methods described herein and is not intended to be limiting.

[0024] Users can interact with the machine using client devices 101, 102, 103, 104, 105, and / or 106. A client device can provide an interface through which a user of the client device interacts with the client device. A client device can also output information to the user through this interface. Although only six client devices are shown in Figure 1, as those skilled in the art will understand, this disclosure can support any number of client devices.

[0025] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices such as portable handheld devices, general-purpose computers (e.g., personal computers and laptops), workstation computers, wearable devices, smartscreen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices may run various types and versions of software applications and operating systems such as MICROSOFT Windows, APPLE iOS, UNIX® operating systems, Linux® or Linux® operating systems (e.g., GOOGLE Chrome OS), or include various mobile operating systems such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include mobile phones, intelligent phones, tablets, and personal digital assistants (PDAs). Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Game systems may include various handheld game devices, internet-enabled game devices, and the like. The client device can run various applications, such as Internet-related applications, communication applications (e.g., email applications), short message service (SMS) applications, and can use various communication protocols.

[0026] Network 110 may be any type of network known to those skilled in the art, and it may use any one of several available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. For example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth®, WIFI), and / or any combination of these and / or other networks.

[0027] Server 120 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX® servers, midrange servers), blade servers, large computers, server clusters, or any other suitable configuration and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures related to virtualization (e.g., one or more flexible pools of virtualized logical memory devices to maintain the server's virtual memory devices). In various embodiments, Server 120 may run one or more services or software applications that provide the functions described below.

[0028] The computing units in server 120 can run one or more operating systems, including any of the above-mentioned operating systems and any commercial server operating systems. Server 120 can also run any one of a variety of additional server applications and / or middle-tier applications, such as an HTTP server, FTP server, CGI server, JAVA® server, or database server.

[0029] In some embodiments, the server 120 may include one or more applications for analyzing and integrating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. The server 120 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0030] In some embodiments, server 120 may be a server in a distributed system or a server incorporating blockchain. Server 120 may be a cloud server, or an intelligent cloud computing server or intelligent cloud host equipped with artificial intelligence technology. A cloud server is a host product in a cloud computing service system and solves the problems of high management difficulty and low business scalability that exist in conventional physical hosts and virtual private server (VPS) services.

[0031] System 100 may also include one or more databases 130. In some embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can be located in various locations. For example, a database used by server 120 may be located locally with server 120, or it may be located away from server 120 and communicate with server 120 via a network or a dedicated connection. The databases 130 may be of various types. In some embodiments, the database used by server 120 may be a relational database. One or more of these databases can store, update, and retrieve data from the databases in response to commands.

[0032] In some embodiments, one or more of the databases 130 may be used by an application to store application data. The database used by the application may be of various types, such as a key-value repository, an object repository, or a general-purpose repository supported by the file system.

[0033] The system 100 in Figure 1 can be configured and operated in various ways so that the various methods and apparatus described in this disclosure can be applied.

[0034] According to one aspect of the present disclosure, a method for determining text processing task similarity is provided. Figure 2 shows a flowchart of method 200 for determining text processing task similarity according to an exemplary embodiment of the present disclosure. As shown in Figure 2, the method includes steps S201 to S203. In S201, a first task, a second task, and a neural network to be trained are determined, the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, the plurality of importance coefficients being used to scale the output values ​​of the respective network modules. In S202, a target operation is performed with the first task and the second task as target tasks to obtain the embedding features of the first task and the second task, respectively, the target operation including step S2021 of training the neural network to be trained using text samples corresponding to the target task and obtaining the trained plurality of importance coefficients, and step S2022 of determining the embedding features of the target task based on the trained plurality of importance coefficients. In S203, the task similarity between the first task and the second task is determined based on the embedding features of the first task and the second task, respectively. Please understand that steps S2021 and S2022 may be substeps of step S202.

[0035] This approach enables the acquisition of embedding features for different text processing tasks with low computational cost and memory overhead by setting multiple importance factors in the neural network to scale the output of multiple network modules, training the neural network using text samples from a text processing task, and then determining the embedding features of the text processing task based on the post-training importance factors. Furthermore, by training a neural network with the same structure using text samples from different text processing tasks, it is possible to acquire task embedding features and obtain more accurate task similarity.

[0036] As mentioned above, text processing tasks refer to specific tasks that require processing and analysis of text data, and can include, for example, text classification, sentiment analysis, naming entity recognition, and summary generation. These text processing tasks can be represented by task embeddings (task-specific vector representations). Task embeddings of different tasks can construct a semantic space to enable a measure of similarity between different text processing tasks.

[0037] In some embodiments, after determining the task similarity between the first task and the second task in step S203, the two tasks can be processed downstream based on their similarity. As shown in Figure 3, the above method may further include step S304, in which a task transition is performed between the first task and the second task in response to the determination that the task similarity between the first task and the second task is greater than a preset similarity. The operation and effects of steps S301 to S303 and their substeps in Figure 3 can be found in the explanation of steps S201 to S203 and their substeps in Figure 2, and are therefore omitted here.

[0038] Task migration refers to improving the performance of another related task by utilizing the knowledge or models of one task. For example, existing news classification task models and data can be used to support a new news classification task, such as migrating from a sports news classification task to an entertainment news classification task, or using an existing movie sentiment analysis model to enhance the effectiveness of product sentiment analysis. This method allows for the rapid, automatic, and accurate acquisition of correlations between tasks, thereby enabling efficient task migration.

[0039] In some embodiments, task migration can be achieved through data augmentation, specifically by using the training set of the source task (Task 1) to enhance the training set of the target task (Task 2), and by increasing the data of related tasks, especially when the target task data is scarce, thereby improving the generalization ability of the model.

[0040] In some embodiments, task transitions can be achieved through a model parameter transfer method, that is, by transferring some or all of the model parameters trained on the source task (first task) to the model of the target task (second task), thereby reducing the training time and resources for the target task.

[0041] In some examples, task transitions can be achieved through collaborative training. In other words, the models for the source task (Task 1) and the target task (Task 2) are trained simultaneously, and the performance of the two tasks is improved by sharing some network structures or parameters.

[0042] As can be seen, a reasonable source task can significantly improve the performance of the target task, while a wrong source task can impair the performance of the target task. Therefore, the selection of source tasks is crucial in task migration. The method of this disclosure allows for the rapid discovery of a large number of source tasks with high similarity to the target task, with low computational cost and memory overhead.

[0043] In some embodiments, after determining the similarity of text processing tasks, further task clustering and task recommendation can be performed.

[0044] Task clustering involves determining the similarity between multiple tasks, including the first and second tasks, and then clustering these tasks based on their similarity, thereby grouping similar tasks together. This is particularly useful for organizing and managing a large number of tasks in multitasking learning or multitasking management scenarios.

[0045] Task recommendation is used on several platforms (e.g., machine learning model sharing platforms) where task embeddings can be used to recommend related tasks or models. In one exemplary implementation, task similarity is determined for several other tasks as second tasks based on the embedding features of a first task that the user is working on, thereby recommending tasks and models similar to the first task and helping the user quickly find relevant resources (datasets of similar tasks, models, etc.).

[0046] It should be understood that the method for determining the similarity of text processing tasks provided in this disclosure can be used in a wider range of scenarios, and is not limited thereto.

[0047] In step S201, the first and second tasks may be two pre-selected text processing tasks for which task similarity needs to be determined. The neural network model to be trained may be any neural network model available for text processing.

[0048] According to some embodiments, the neural network to be trained is a Transformer architecture, and the network modules include multiple self-attention modules and multiple feedforward neural network modules.

[0049] In a Transformer architecture neural network, each layer primarily consists of two parts: a multi-head attention module and a feedforward neural network module. The multi-head attention mechanism can be formalized as follows:

number

[0050] A feedforward neural network module can be formalized as follows:

number

[0051] The learnable importance coefficient m for each self-attention head H can be assigned, and the learnable importance coefficient m F is assigned to each feed-forward neural network.

Number

[0052] After training, the above-mentioned importance coefficient can scale the output value of the corresponding network module, thereby realizing retaining or enlarging the output value of the important network module and shrinking the output value of the network module with little influence on the output.

[0053] In step S202, the first task and the second task are respectively used as target tasks to perform target operations, and the respective embedding features of the first task and the second task can be obtained. As described above, the target operation can include step S2021 and step S2022.

[0054] Before executing step S2021, a training set or dataset corresponding to the target task may be obtained, which can include a plurality of text samples. The text sample can include text data and can also include the corresponding ground truth tag.

[0055] In step S2021, text data from a text sample is input to the neural network to be trained, and the text processing result output by the neural network to be trained can be obtained. Furthermore, the parameters of the neural network to be trained can be adjusted based on the difference between the text processing result and the actual tab, and this includes learnable parameters in multiple network modules and multiple importance coefficients. After the training of the neural network to be trained is complete, the trained multiple importance coefficients can be obtained.

[0056] In step S2022, the multiple trained importance coefficients can be directly identified as embedding features for the target task.

[0057] In some embodiments, the neural network to be trained is a large language model. For large language models, the computational and memory costs of obtaining task embeddings using conventional methods are very high. However, using the method of this disclosure, the amount of data that needs to be stored is only the number of multi-head attention modules and the number of feedforward neural networks in the large language model. In some embodiments, the importance coefficient can be a Boolean variable (described below) to further reduce the amount of data that needs to be stored.

[0058] According to several examples, the initial values ​​of multiple importance coefficients are obtained by randomly initializing them. By training a neural network using text samples from different tasks, the multiple randomly initialized importance coefficients can be converged to combine the importance coefficients corresponding to the different tasks.

[0059] In some embodiments, as shown in Figure 4, step S2021 involves training a neural network to be trained using text samples corresponding to the target task and obtaining multiple trained importance coefficients, which includes steps S401 to S403. In step S401, a first loss value is determined based on the multiple importance coefficients, and the first loss value has a positive correlation with the absolute values ​​of the multiple importance coefficients. In step S402, the text processing results output by the neural network to be trained based on the text samples are obtained, and a second loss value is determined based on the text processing results, and the second loss value is used to evaluate the text processing results. In step S403, the multiple importance coefficients and the learnable parameters of multiple network modules are adjusted based on the first and second loss values.

[0060] Given that neural networks are sparse, meaning that the network modules that play a crucial role in a neural network are sparse, the multiple importance coefficients that are trained should also be sparse. To ensure the sparseness of the importance coefficients, a regularization term, i.e., a first loss value, can be added to the training goal.

number

number

number

[0061] In some embodiments, as shown in Figure 5, step S2021 may include steps S504 to S505 to train a neural network to be trained using text samples corresponding to the target task and to obtain multiple trained importance coefficients. In step S504, after adjusting the parameter for a predetermined number of iterations, the current multiple importance coefficients are converted into multiple instruction values ​​based on a predetermined threshold. In step S505, after adjusting the parameter for a succession of predetermined number of iterations, training of the neural network to be trained is stopped in response to the determination that the changes in the multiple instruction values ​​satisfy a predetermined rule. Note that the operation of steps S501 to S503 in Figure 5 can be explained by referring to the above explanation for steps S401 to S403 and will not be explained here.

[0062] In some embodiments, the importance coefficient can be converted into an indicator value. These indicators can converge early (within one round, i.e., beyond a single traverse of a complete training set). Therefore, an early stopping strategy can be employed, allowing the model to be stopped at an early training point according to the rules above. In this case, the neural network itself may not yet have converged, meaning it has not yet fully completed the corresponding target task, but the indicators obtained by transforming the importance coefficients have already converged.

[0063] The pre-set number of iterations is sometimes called a mini-epoch or micro-epoch, and its value can be set as needed and is not limited here. In step S505, after adjusting the parameters of multiple consecutive N pre-set number of iterations, training of the neural network to be trained is stopped in response to the confirmation that the changes in multiple indicator values ​​satisfy the pre-set rules. Note that the value of N can be set as needed and is not limited here.

[0064] In some embodiments, the indicated value can take the value of 0 or 1, and importance coefficients exceeding a preset threshold can be converted to 1, while importance coefficients not exceeding a preset threshold can be converted to 0.

[0065] In some embodiments, all of the indicator values ​​are Boolean variables, and the pre-defined rule includes that, after parameter adjustment for a set number of consecutive pre-defined iterations, the number of changing indicator values ​​among the indicator values ​​does not exceed the pre-defined parameter.

[0066] In some embodiments, training can be stopped if the mask change over several consecutive mini-epochs does not exceed a single fixed parameter γ. It is understood that the value of γ can be set as needed and is not limited here.

[0067] In some embodiments, determining the embedding features of the target task based on the trained importance coefficients in step S2022 may include determining the multiple instruction values ​​obtained in the final transformation as the embedding features of the target task.

[0068] In some embodiments, the instruction values ​​obtained from the final transformation of importance coefficients corresponding to multiple self-attention mechanism modules are concatenated with the instruction values ​​obtained from the final transformation of importance coefficients corresponding to multiple feedforward neural network modules to obtain embedding features for the target task.

[0069] In some embodiments, determining the task similarity between the first task and the second task based on their respective embedding features in step S203 may include determining the task similarity between the first task and the second task based on the number of indicator values ​​that are identical in position and have the same value in the embedding features of the first task and the embedding features of the second task.

[0070] Identical indication values ​​in the same position correspond to the same network module (self-attention head or feedforward neural network), and identical values ​​mean that the network module is not important in either the first or second task. Therefore, accurate task similarity can be obtained simply and quickly using the method described above.

[0071] In another aspect of the present disclosure, an apparatus for determining text processing task similarity is provided. As shown in Figure 6, the apparatus 600 includes a first determination unit 610 configured to determine a first task, a second task, and a neural network to be trained, wherein the neural network to be trained includes a plurality of network modules and a plurality of importance coefficients corresponding to the plurality of network modules, and the plurality of importance coefficients are used to scale the output values ​​of the respective network modules; an embedding feature acquisition unit 620 configured to perform target operations with the first task and the second task as target tasks to obtain embedding features for the first task and the second task, respectively, and configured to train the neural network to be trained using text samples corresponding to the target tasks and to obtain the trained plurality of importance coefficients; and a first determination subunit 624 configured to determine the embedding features of the target task based on the trained plurality of importance coefficients; and a second determination unit 630 configured to determine the task similarity between the first task and the second task based on the embedding features of the first task and the second task, respectively.

[0072] For ease of understanding, the operation and effects of units 610 to 630 and their subunits in apparatus 600 can be found in the above-described steps S201 to S203 and their substeps, and will not be described here.

[0073] According to some embodiments, a device for determining the similarity of text processing tasks may further include a task transition unit configured to perform a task transition between a first task and a second task in response to the determination that the task similarity between a first task and a second task is greater than a preset similarity.

[0074] According to some embodiments, task transition includes at least one of the following: strengthening the training set for the second task using the training set for the first task; transferring at least some of the model parameters in the neural network trained for the first task to the neural network for the second task; and training the neural network for the first task and the neural network for the second task simultaneously, wherein the neural network for the first task and the neural network for the second task share some structure or parameters.

[0075] In some embodiments, the neural network to be trained is a Transformer architecture, and the network modules may include multiple self-attention modules and multiple feedforward neural network modules.

[0076] In some embodiments, the neural network to be trained may be a large language model.

[0077] In some embodiments, the initial values ​​of multiple importance coefficients may be obtained by randomly initializing them.

[0078] In some embodiments, the training subunit may include a second determination subunit configured to determine a first loss value based on a plurality of importance coefficients, such that the first loss value has a positive correlation with the absolute values ​​of the plurality of importance coefficients; an acquisition subunit configured to acquire the text processing results output by the neural network to be trained based on text samples, and to determine a second loss value based on the text processing results, such that the second loss value is used to evaluate the text processing results; and a parameter adjustment subunit configured to adjust the plurality of importance coefficients and the learnable parameters of the plurality of network modules based on the first and second loss values.

[0079] In some embodiments, the training subunit may include a transformation subunit configured to convert a set number of current importance coefficients into a set number of instruction values ​​based on a set threshold after parameter tuning for a set number of pre-set iterations, and an early stop subunit configured to stop training the neural network to be trained in response to changes in the number of instruction values ​​satisfying a set rule after parameter tuning for a set number of consecutive pre-set iterations.

[0080] In some embodiments, all of the indicator values ​​are Boolean variables, and the pre-defined rule may include ensuring that, after a series of pre-defined parameter adjustments, the number of changing indicator values ​​among the indicator values ​​does not exceed the pre-defined parameter.

[0081] According to some embodiments, the first deterministic subunit may include a third deterministic subunit configured to determine a plurality of indicator values ​​obtained in the final transformation as embedding features of the target task.

[0082] According to some embodiments, the second determinative unit may include a fourth determinative subunit configured to determine the task similarity between the first task and the second task based on the number of indicator values ​​that are in the same position and have the same value in the embedding features of the first task and the embedding features of the second task.

[0083] In the proposed technology described herein, all processing of relevant user personal information, including collection, storage, use, processing, transmission, provision, and disclosure, complies with the provisions of relevant laws and regulations and does not violate public order and morals.

[0084] Embodiments of this disclosure further provide electronic devices, readable storage media, and computer program products.

[0085] Next, with reference to Figure 7, a block diagram of the configuration of an electronic device 700 that functions as a server or client of the present disclosure is described, which is an example of hardware equipment applicable to each aspect of the present disclosure. The electronic device represents various forms of digital electronic computers, e.g., laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, large computers, and other suitable computers. The electronic device may further represent various forms of mobile devices, e.g., personal digital processing devices, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are illustrative and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0086] As shown in Figure 7, the electronic device 700 includes a computing unit 701 that can perform various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data necessary for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0087] Multiple components of the electronic device 700, including an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709, are connected to an I / O interface 705. The input unit 706 may be any type of device capable of inputting information into the electronic device 700, and may receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device, including but not limited to a mouse, keyboard, touchscreen, trackboard, trackball, lever, microphone, and / or remote control. The output unit 707 may be any type of device capable of presenting information, including but not limited to a display, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 708 may include, but is not limited to, a magnetic disk or an optical disk. The communication unit 709 enables the electronic device 700 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMAX devices, cellular communication devices and / or similar devices.

[0088] The computing unit 701 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs each of the methods, processes, and / or operations described above. For example, in some embodiments, these methods, processes, and / or operations can be implemented as computer software programs tangibly contained in a machine-readable medium such as a storage unit 708. In some embodiments, some or all of the computer program may be loaded and / or installed into the electronic device 700 via ROM 702 and / or communication unit 709. Once the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods, processes, and / or operations described above can be performed. Alternatively, in other embodiments, the computing unit 701 is configured to perform these methods, processes, and / or operations in any other suitable manner (e.g., by firmware).

[0089] Various embodiments of the systems and technologies described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Program code for implementing the methods of this disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, it performs the functions / operations specified in the flowcharts and / or block diagrams. The program code may be executed entirely by machine, partially by machine, partially by machine and partially by remote machine as a standalone software package, or entirely by remote machine or server.

[0091] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or stores a program used in or used in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any appropriate combination thereof. More specific examples of machine-readable storage media include an electrical connection by one or more leads, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage equipment, magnetic storage equipment, or any appropriate combination thereof.

[0092] To provide user interaction, the computer may implement the systems and technologies described herein, the computer comprising a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitoring monitor), and a keyboard and pointing device (e.g., a mouse or trackball), the user may input to the computer via the keyboard and pointing device. Other types of devices may further provide user interaction, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and the computer may receive input from the user in any form (including sound input, voice input, or tactile input).

[0093] The systems and technologies described herein may be implemented in computing systems including backstage components (e.g., as data servers), computing systems including middleware components (e.g., application servers), computing systems including front-end components (e.g., user computers having a graphical user interface or web browser, through which users can interact with embodiments of those systems and technologies), or computing systems consisting of any combination of these backstage components, middleware components, or front-end components. The components of the system may be interconnected by digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local networks (LANs), wide area networks (WANs), the internet, and blockchain networks.

[0094] A computer system may include a client and a server. The client and server are generally geographically distant from each other and typically interact via a communication network. The client-server relationship is created by running computer programs on corresponding computers that have a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0095] It should be understood that the steps may be reordered, added, or deleted using the various forms of flows described above. For example, each step described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the technical proposal disclosed herein achieves the desired result.

[0096] While the embodiments or examples of this disclosure are described with reference to the drawings, it should be understood that the above-described methods, systems, and apparatus are merely illustrative embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but is limited only by the authorized claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by their equivalent elements. Furthermore, each step may be performed in an order different from the order described herein. Moreover, various elements of the embodiments or examples may be combined in various ways. Importantly, as technology advances, many of the elements described herein can be replaced by equivalent elements appearing later in this disclosure.

Claims

1. A method for determining the similarity of text processing tasks, wherein the method is performed by a computer. The first task, the second task, and the neural network to be trained are determined, and the neural network to be trained includes multiple network modules and multiple importance factors corresponding to the multiple network modules, and the multiple importance factors are used to scale the output values ​​of the respective network modules. This includes performing a target operation with the first task and the second task as target tasks, respectively, to obtain the embedding features of the first task and the second task, wherein the target operation is The process involves training the neural network to be trained using text samples corresponding to the aforementioned target task, and obtaining multiple importance coefficients from the trained network. Based on the trained multiple importance factors, the embedding features of the target task are determined, A method for determining text processing task similarity, which includes determining the task similarity between the first task and the second task based on the embedding features of the first task and the second task, respectively.

2. Training the neural network to be trained using text samples corresponding to the aforementioned target task, and obtaining multiple trained importance coefficients, Based on the aforementioned multiple importance coefficients, a first loss value is determined, and the first loss value has a positive correlation with the absolute values ​​of the aforementioned multiple importance coefficients. The neural network to be trained obtains the text processing result output based on the text sample, determines a second loss value based on the text processing result, and uses the second loss value to evaluate the text processing result. The method according to claim 1, further comprising adjusting the plurality of importance coefficients and the learnable parameters of the plurality of network modules based on the first loss value and the second loss value.

3. Training the neural network to be trained using text samples corresponding to the aforementioned target task, and obtaining multiple trained importance coefficients, After adjusting the parameter for the number of pre-set iterations, the current multiple importance coefficients are converted into multiple indicator values ​​based on a pre-set threshold, The method according to claim 2, further comprising stopping the training of the neural network to be trained in response to the determination that, after adjusting the parameters for a plurality of consecutive preset iterations, the changes in a plurality of indicator values ​​satisfy a preset rule.

4. The method according to claim 3, wherein all of the aforementioned multiple indicator values ​​are Boolean variables, and the pre-set rule includes parameter adjustments for a series of predetermined iterations, after which the number of changing indicator values ​​among the multiple indicator values ​​does not exceed the predetermined parameter.

5. Based on the aforementioned trained importance factors, determining the embedding features of the target task is: The method according to claim 3, comprising determining a plurality of instruction values ​​obtained in the final transformation as embedding features of the target task.

6. Determining the task similarity between the first task and the second task based on the respective embedding features of the first task and the second task is: The method according to claim 5, comprising determining the task similarity between the first task and the second task based on the number of indicator values ​​that are in the same position and have the same value in the embedding features of the first task and the embedding features of the second task.

7. The method according to claim 1, wherein the neural network to be trained is a Transformer architecture, and the plurality of network modules include a plurality of self-attention modules and a plurality of feedforward neural network modules.

8. The method according to claim 7, wherein the neural network to be trained is a large language model.

9. The method according to claim 1, wherein the initial values ​​of the plurality of importance coefficients are obtained by randomly initializing them.

10. The method according to claim 1, further comprising performing a task transition between the first task and the second task in response to the determination that the task similarity between the first task and the second task is greater than a preset similarity.

11. The aforementioned task transition is The training set for the second task is reinforced using the training set for the first task, Transferring at least some of the model parameters in the neural network trained for the first task to the neural network for the second task, The method according to claim 10, comprising at least one of training a neural network for the first task and a neural network for the second task simultaneously, wherein the neural network for the first task and the neural network for the second task share some structure or parameters.

12. A device for determining the similarity of text processing tasks, A first determination unit is configured to determine the first task, the second task, and the neural network to be trained, wherein the neural network to be trained includes multiple network modules and multiple importance factors corresponding to the multiple network modules, and each of the multiple importance factors is used to scale the output value of the corresponding network module. The system is configured to perform target operations with the first task and the second task as target tasks, in order to obtain the embedding features of the first task and the second task, respectively. A training subunit configured to train the neural network to be trained using text samples corresponding to the target task, and to obtain multiple trained importance coefficients, An embedding feature acquisition unit includes a first determinative subunit configured to determine the embedding features of the target task based on the trained multiple importance coefficients, A device for determining text processing task similarity, comprising a second determination unit configured to determine the task similarity between the first task and the second task based on the embedding features of the first task and the second task, respectively.

13. The aforementioned training subunit is A second determinative subunit is configured to determine a first loss value based on the aforementioned multiple importance coefficients, and the first loss value is configured to have a positive correlation with the absolute values ​​of the aforementioned multiple importance coefficients. An acquisition subunit is configured to acquire the text processing result output by the neural network to be trained based on the text sample, and to determine a second loss value based on the text processing result, and to use the second loss value to evaluate the text processing result, The apparatus according to claim 12, further comprising a parameter tuning subunit configured to adjust the plurality of importance coefficients and the learnable parameters of the plurality of network modules based on the first loss value and the second loss value.

14. The aforementioned training subunit is A conversion subunit configured to convert multiple current importance coefficients into multiple indicator values ​​based on a preset threshold after adjusting the parameter for a preset number of iterations, The apparatus according to claim 13, further comprising: an early stop subunit configured to stop training the neural network to be trained in response to the determination that, after parameter adjustments for a plurality of consecutive preset iterations, changes in a plurality of indicator values ​​satisfy a preset rule.

15. The apparatus according to claim 14, wherein all of the aforementioned multiple indicator values ​​are Boolean variables, and the pre-set rule includes parameter adjustments for a succession of the aforementioned pre-set number of iterations, after which the number of changing indicator values ​​among the multiple indicator values ​​does not exceed the pre-set parameter.

16. The aforementioned first definitive subunit is, The apparatus according to claim 14, comprising a third determinative subunit configured to determine a plurality of instruction values ​​obtained in the final conversion as embedding features of the target task.

17. The aforementioned second confirmed unit is, The apparatus according to claim 16, comprising a fourth determining subunit configured to determine the task similarity between the first task and the second task based on the number of indicator values ​​that are in the same position and have the same value in the embedding features of the first task and the embedding features of the second task.

18. The apparatus according to any one of claims 12 to 17, wherein the neural network to be trained is a Transformer architecture, and the plurality of network modules include a plurality of self-attention modules and a plurality of feedforward neural network modules.

19. The apparatus according to claim 18, wherein the neural network to be trained is a large language model.

20. The apparatus according to any one of claims 12 to 17, wherein the initial values ​​of the plurality of importance coefficients are obtained by random initialization.

21. The apparatus according to any one of claims 12 to 17, further comprising a task transition unit configured to perform a task transition between the first task and the second task in response to the determination that the task similarity between the first task and the second task is greater than a preset similarity.

22. The aforementioned task transition is The training set for the second task is reinforced using the training set for the first task, Transferring at least some of the model parameters in the neural network trained for the first task to the neural network for the second task, The apparatus according to claim 21, comprising at least one of training a neural network for the first task and a neural network for the second task simultaneously, wherein the neural network for the first task and the neural network for the second task share some structure or parameters.

23. Electronic equipment, said electronic equipment is At least one processor, Includes memory communicated to at least one processor, An electronic device wherein the memory stores instructions that can be executed by the at least one processor, and the execution of the instructions by the at least one processor causes the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-temporary computer-readable storage medium storing computer instructions, wherein the computer instructions cause a computer to execute the method according to any one of claims 1 to 11.

25. A computer program, wherein, when executed by a processor, the computer program performs the method according to any one of claims 1 to 11.

Images