A method for generating a topic text and a related device

Abstract

The present application discloses a method for generating a topic text and a related device, which relates to the field of computer technology, including: obtaining a target cooperative topic vector to be converted into a text, using a migration fine-tuning layer in a pre-trained text generation model to process the target cooperative topic vector into a target multi-angle feature vector in a target mapping space, and using a base large language model in the text generation model to process the target multi-angle feature vector into a topic text corresponding to the target cooperative topic vector. The present application simultaneously trains the parameters of the migration fine-tuning layer and the parameters of the preset layer based on the training data, and by pre-training a large number of parameters of the preset layer in the base large language model, it can be ensured that after the target cooperative topic vector is converted into a target multi-angle feature vector in a target mapping space, the target multi-angle feature vector can be accurately understood and decoded by the base large language model, thereby improving the readability and accuracy of the topic text finally generated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating a subject text and a related device. Background Art

[0002] In order to produce high-value results in scientific research, researchers usually need to find other researchers to cooperate with in order to jointly research new scientific research results. In order to help researchers quickly find suitable scientific research collaborators, the collaborator prediction model (the collaborator prediction model is an implementation in CN118966440A) can be used to predict the cooperation relationship between the target scientific research author and other scientific research authors to obtain the collaborators of the target scientific research author. In addition, the collaborator prediction model can also output the cooperation time and cooperation theme between the target scientific research author and the collaborator.

[0003] Since the cooperation topic output by the collaborator prediction model is in the form of a vector that can only be understood by computers, it is necessary to decode the cooperation topic vector through the decoder in the existing model (such as the large language model) to obtain the cooperation topic text. However, the mapping space where the cooperation topic vector is located may be different from the mapping space that can be processed by the existing model, resulting in poor readability and accuracy of the cooperation topic text output by the existing model. Summary of the invention

[0004] In view of the above problems, the present application provides a method and a related device for generating a topic text, so as to achieve the purpose of converting the cooperation topic vector into a topic text with high readability and high accuracy. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for generating a subject text, comprising:

[0006] Obtaining a target cooperative topic vector for text conversion;

[0007] Using a transfer fine-tuning layer in a pre-trained text generation model, the target cooperative topic vector is processed into a target multi-angle feature vector in a target mapping space, wherein the target mapping space refers to a mapping space processable by a base large language model in the text generation model;

[0008] The target multi-angle feature vector is processed into the topic text corresponding to the target cooperative topic vector by using the base large language model in the text generation model, wherein the parameters of the migration fine-tuning layer and the parameters of the preset layer in the base large language model are obtained by training with the training text semantic vector annotated with the training text label as the training data, and the training text semantic vector and the target cooperative topic vector are in the same mapping space.

[0009] In a possible implementation, the training process of the text generation model includes:

[0010] Obtaining a text semantic vector of the training text as the training text semantic vector;

[0011] Based on the training text and the semantic vector of the training text, training parameters of the initial migration fine-tuning layer and parameters of the preset layer in the initial base large language model, obtaining a post-training migration fine-tuning layer and a post-training base large language model, wherein the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is a first preset value at the time of initial construction;

[0012] Determine whether the topic text output by the concatenated model consisting of the post-training transfer fine-tuning layer and the post-training base large language model meets the text generation requirements;

[0013] If yes, the concatenation model is determined as the text generation model;

[0014] If not, the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is adjusted, and the parameters of the initial migration fine-tuning layer trained based on the training text and the training text semantic vector and the parameters of the preset layer in the initial base large language model are returned.

[0015] In a possible implementation, obtaining the text semantic vector of the training text includes:

[0016] Extracting the title, abstract and keywords of the training text;

[0017] Based on the title, the abstract and the keywords, generating a prompt sentence corresponding to the training text;

[0018] Align the prompt sentence corresponding to the training text to the left, and add an end mark to the end of the prompt sentence to obtain a processed prompt sentence corresponding to the training text;

[0019] A text semantic vector of the training text is obtained according to the processed prompt sentence.

[0020] In a possible implementation, the training of the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model based on the training text and the semantic vector of the training text includes:

[0021] Inputting the training text semantic vector into the initial migration fine-tuning layer to obtain a training multi-angle feature vector output by the initial migration fine-tuning layer, wherein each of the sub-migration fine-tuning layers included in the initial migration fine-tuning layer is used to map the training text semantic vector into a sub-vector, and the training multi-angle feature vector is obtained by concatenating the sub-vectors mapped by all the sub-migration fine-tuning layers;

[0022] Left-aligning the training text, and adding a preset prompt word to the head of the left-aligned training text to obtain a pre-processed training text;

[0023] Processing the preprocessed training text into an initial word unit sequence using a word unit device in the initial base large language model;

[0024] The last word in the initial word sequence is removed and the end is padded with a blank character to obtain a first aligned word sequence;

[0025] Processing the first aligned word-unit sequence into an embedding vector using an encoder in the initial base large language model;

[0026] Concatenate the embedding vector and the training multi-angle feature vector forward and backward to obtain a concatenated vector;

[0027] The first word in the initial word-unit sequence is removed, and the end is padded with the blank character to obtain a second aligned word-unit sequence;

[0028] Adding a second preset value of the target number to the head of the second aligned word sequence, and modifying the blank characters in the second aligned word sequence to the second preset value, to obtain a gold standard sequence, wherein the target number is equal to the number of sub-transfer fine-tuning layers included in the transfer fine-tuning layer;

[0029] Determine a mask sequence corresponding to the splicing vector, wherein the mask value at the position where the placeholder in the splicing vector is located is a first value, and the mask value at the non-placeholder position is a second value, and the placeholder includes the blank character and the end character;

[0030] The parameters of the initial transfer fine-tuning layer and the parameters of the preset layer are trained based on the mask sequence, the concatenated vector and the gold standard sequence to obtain the trained transfer fine-tuning layer and the trained base large language model.

[0031] In a possible implementation, the training of the parameters of the initial migration fine-tuning layer and the parameters of the preset layer based on the mask sequence, the splicing vector and the gold standard sequence includes:

[0032] Inputting the concatenated vector into the initial base large language model to obtain the training topic text output by the initial base large language model;

[0033] Calculating a total loss value based on the training subject text, the mask sequence, and the gold standard sequence;

[0034] Release the parameters of the preset layer in the initial base large language model;

[0035] Parameters of the initial migration fine-tuning layer and parameters of the preset layer are trained based on the total loss value.

[0036] In a possible implementation, the preset layers are the last N layers of the base large language model, and the size of N is determined according to the generation evaluation index.

[0037] In a possible implementation, the first preset value is an empirical value determined based on a parameter amount of the initial base large language model and a parameter amount of the sub-migration fine-tuning layer.

[0038] In a possible implementation, the sub-migration fine-tuning layer uses a multi-layer linear perceptron or a Transformer encoder.

[0039] A second aspect of the present application provides a device for generating a subject text, comprising:

[0040] A vector acquisition module, used to acquire a target cooperative topic vector to be converted into text;

[0041] A space mapping module, used to process the target cooperative topic vector into a target multi-angle feature vector under a target mapping space using a transfer fine-tuning layer in a pre-trained text generation model, wherein the target mapping space refers to a mapping space processable by a base large language model in the text generation model;

[0042] A text prediction module is used to use the base large language model in the text generation model to process the target multi-angle feature vector into a topic text corresponding to the target cooperative topic vector, wherein the parameters of the migration fine-tuning layer and the parameters of the preset layer in the base large language model are trained using the training text semantic vector annotated with the training text label as training data, and the training text semantic vector and the target cooperative topic vector are in the same mapping space.

[0043] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for generating a subject text of the first aspect or any implementation of the first aspect.

[0044] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0045] The memory is used to store computer programs;

[0046] The processor is used to execute the computer program so that the electronic device can implement the method for generating the subject text of the first aspect or any implementation manner of the first aspect.

[0047] The fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the method for generating the subject text of the above-mentioned first aspect or any implementation method of the first aspect.

[0048] By means of the above technical solution, the method for generating the subject text provided by the present application obtains the target cooperative subject vector to be converted into text, uses the migration fine-tuning layer in the pre-trained text generation model to process the target cooperative subject vector into a target multi-angle feature vector under the target mapping space, and uses the base large language model in the text generation model to process the target multi-angle feature vector into the subject text corresponding to the target cooperative subject vector. It can be seen that in order to be able to convert the target cooperative subject vector into a target mapping space that can be processed by the base large language model, the present application releases the parameters of the preset layer in the base large language model, so as to simultaneously train the parameters of the migration fine-tuning layer and the parameters of the preset layer based on the training data. By training a large number of parameters of the preset layer in the base large language model, it can be ensured that after the target cooperative subject vector is converted into the target multi-angle feature vector under the target mapping space, the target multi-angle feature vector can be accurately understood and decoded by the base large language model, thereby improving the readability and accuracy of the subject text finally generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0050] Figure 1 A schematic diagram of a system architecture provided for this application;

[0051] Figure 2 A schematic diagram of an optional hardware structure of the terminal 100 provided in this application;

[0052] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0053] Figure 4 A flowchart of a method for generating a subject text provided in this application;

[0054] Figure 5 Schematic diagram of the process of determining the target cooperation theme vector provided for this application;

[0055] Figure 6 A schematic diagram of the structure of the text generation model provided for this application;

[0056] Figure 7 A schematic diagram of the training process of a text generation model provided in this application;

[0057] Figure 8 A schematic diagram of the process of initial migration fine-tuning layer and initial base large language model pre-training provided in this application;

[0058] Fig. 9 A schematic diagram of the structure of a device for generating a subject text provided in this application;

[0059] Fig.10 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0061] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0062] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment containing a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0063] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0064] Among them, an application can be installed on the terminal 100, and the above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0065] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0066] Next describe Figure 1 The product form of the mid-terminal 100;

[0067] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0068] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0069] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0070] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0071] Among them, the input device 132 can receive input data and the like.

[0072] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0073] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0074] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0075] Among them, the memory 120 can be used to store software codes related to the method for generating the topic text, the processor 170 can execute the steps of the method for generating the topic text, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0076] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile Communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0077] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0078] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0079] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0080] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0081] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0082] Next describe Figure 1 The product form of the server 200;

[0083] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0084] The bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0086] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0087] The memory 204 may be used to store software codes related to the method for generating the subject text, and the processor 202 may execute the steps of the method for generating the subject text of the chip, and may also schedule other units to implement corresponding functions.

[0088] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, DSPs, microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0089] The present application provides a method for generating a topic text. The method for generating a topic text of an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0090] Reference Figure 4 , Figure 4 A flowchart of a method for generating a topic text provided in an embodiment of the present application, the method may include:

[0091] Step S401: Obtain a target collaborative topic vector for text conversion.

[0092] See also Figure 5 , is a schematic diagram of the process of determining the target cooperation theme vector provided in this application. Figure 5 , the target cooperation theme vector is the vector representing the cooperation theme output by the collaborator prediction model. In more detail, the original text is processed by the text semantic embedding model to obtain the target text semantic vector, and the target text semantic vector is further processed by the collaborator prediction model to obtain the target cooperation theme vector. Since the collaborator prediction model does not transform the target text semantic vector in the mapping space, the target text semantic vector and the target cooperation theme vector are in the same mapping space.

[0093] Optionally, the above text semantic embedding model is an all-mpnet-base-v2 model, which is a model that maps sentences and paragraphs to a vector space. It can convert a given input text into a sentence vector that captures semantic information. These sentence vectors can be used for tasks such as information retrieval, clustering, or sentence similarity.

[0094] Optionally, the original text may be a published text of various types such as a paper or patent. Of course, the original text may also be other texts, which are not specifically limited in this application. The target cooperation theme vector output by the collaborator prediction model is information that can only be recognized by a computer, and it is also necessary to convert it into a readable cooperation theme text for the user to understand.

[0095] Step S402: Use the transfer fine-tuning layer in the pre-trained text generation model to process the target collaborative topic vector into a target multi-angle feature vector in the target mapping space.

[0096] The target mapping space refers to the mapping space that can be processed by the base large language model in the above text generation model.

[0097] Following the introduction in the background technology, the mapping space that can be processed by the current model may be different from the mapping space where the target cooperation theme vector is located. For example, the mapping space where the target cooperation theme vector is located is a 1024-dimensional space, while the mapping space that can be processed by the current model may be a 512-dimensional space. Due to the different mapping spaces, the current model cannot well understand the meaning represented by the target cooperation theme vector, and thus cannot accurately decode it. According to experiments, the current model is used to decode the target cooperation theme vector, and the decoded text is extremely incoherent and has poor readability.

[0098] In order to decode the cooperation theme text that meets the requirements of readability and accuracy, the embodiment of the present application provides a pre-trained text generation model, such as Figure 6 , which is a structural diagram of the text generation model provided in this application, and the text generation model is composed of a transfer fine-tuning layer and a base large language model, wherein the transfer fine-tuning layer can serve as a bridge for communication between the target cooperative topic vector and the base large language model, so that the target cooperative topic vector can be mapped to the target mapping space that can be understood and processed by the base large language model.

[0099] In order to enable the target cooperative topic vector to be decoded by the base large language model, the present embodiment may first input the target cooperative topic vector into the transfer fine-tuning layer so that the target cooperative topic vector can be mapped to the target mapping space through the transfer fine-tuning layer. For the convenience of subsequent introduction, the vector obtained by mapping the target cooperative topic vector to the target mapping space is recorded as the target multi-angle feature vector.

[0100] Step S403: Use the base large language model in the text generation model to process the target multi-angle feature vector into a topic text corresponding to the target cooperative topic vector.

[0101] Optionally, the above-mentioned base large language model can be a causal language model (Casual Language Model, CasualLM) model, and the base large language model can use the next token predication algorithm to process the target multi-angle feature vector into a topic text corresponding to the target cooperative topic vector.

[0102] Among them, the parameters of the migration fine-tuning layer and the parameters of the preset layer in the base large language model are trained using the training text semantic vectors annotated with training text labels as training data, and the training text semantic vectors and the target cooperation topic vectors are in the same mapping space.

[0103] The above training text semantic vector corresponds to the target text semantic vector in the previous text, and the training text semantic vector is obtained by processing the training text through a text semantic embedding model.

[0104] It is worth noting that the present application uses the training text semantic vectors with annotated training text labels as training data, rather than using the training cooperation topic vectors with annotated training text labels (corresponding to the target cooperation topic vectors in the previous text) as training data. This is because the collaborator prediction model does not undergo a mapping space conversion in the process of processing the training text semantic vectors into training cooperation topic vectors, so that the training cooperation topic vectors and the training text semantic vectors are in the same mapping space. For this reason, there will be no significant difference between the model trained by using the training cooperation topic vectors with annotated training text labels as training data and the model trained by using the training text semantic vectors with annotated training text labels as training data. However, processing the training text semantic vectors into training cooperation topic vectors requires additional steps, which reduces the efficiency of model training.

[0105] The method for generating a topic text provided in the present application obtains a target cooperative topic vector to be converted into text, uses a migration fine-tuning layer in a pre-trained text generation model to process the target cooperative topic vector into a target multi-angle feature vector in a target mapping space, and uses the base large language model in the text generation model to process the target multi-angle feature vector into a topic text corresponding to the target cooperative topic vector. It can be seen that in order to be able to convert the target cooperative topic vector into a target mapping space that can be processed by the base large language model, the present application releases the parameters of the preset layer in the base large language model, so as to simultaneously train the parameters of the migration fine-tuning layer and the parameters of the preset layer based on the training data. By training a large number of parameters of the preset layer in the base large language model, it can be ensured that after the target cooperative topic vector is converted into a target multi-angle feature vector in the target mapping space, the target multi-angle feature vector can be accurately understood and decoded by the base large language model, thereby improving the readability and accuracy of the topic text finally generated.

[0106] In order to enable those skilled in the art to better understand the above-mentioned text generation model, the training process of the text generation model is introduced in detail below.

[0107] See also Figure 7 , which is a schematic diagram of the training process of a text generation model provided in this application.

[0108] Step S501: Obtain a text semantic vector of a training text as a training text semantic vector.

[0109] Optionally, the process of "obtaining the text semantic vector of the training text" can be implemented through the above-mentioned text semantic embedding model. More specifically, the process of "obtaining the text semantic vector of the training text" may include: extracting the title, abstract and keywords of the training text, generating a prompt sentence corresponding to the training text based on the title, abstract and keywords, left-aligning the prompt sentence corresponding to the training text, and adding an end mark to the end of the prompt sentence to obtain the processed prompt sentence corresponding to the training text, and obtaining the text semantic vector of the training text according to the processed prompt sentence.

[0110] It should be noted that the above process of generating prompt sentences based on the extracted title, abstract and keywords is only an example and is not intended to limit the present application. In actual application scenarios, this embodiment can also extract more or less information to generate prompt sentences.

[0111] Optionally, the process of "obtaining a text semantic vector of the training text according to the processed prompt sentence" can be implemented by the above-mentioned text semantic embedding model, for example, by the all-mpnet-base-v2 model.

[0112] Step S502: train the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model based on the training text and the training text semantic vector to obtain the post-training migration fine-tuning layer and the post-training base large language model.

[0113] Specifically, in this embodiment, a large language model may be selected as the initial base large language model, and optionally, the initial base large language model may be a CasualLM model. In addition, an initial migration fine-tuning layer may be constructed as a prefix of the initial base large language model.

[0114] The above-mentioned initial migration fine-tuning layer includes at least one sub-migration fine-tuning layer. Optionally, when the initial migration fine-tuning layer includes multiple sub-migration fine-tuning layers, the multiple sub-migration fine-tuning layers have the same structure but independent parameters. In this embodiment, multiple sub-migration fine-tuning layers with the same structure but independent parameters can form a multi-head mapping module to capture features in the semantic vector of the training text from multiple different angles.

[0115] Optionally, the sub-migration fine-tuning layer may adopt a multilayer linear perceptron (MLP) or a Transformer encoder. Of course, the sub-migration fine-tuning layer may also be other, which is not specifically limited in this application.

[0116] In this embodiment, the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is a first preset value at the time of initial construction. Here, the first preset value is a hyperparameter, which is represented by prefix_lenth in this embodiment.

[0117] Optionally, the first preset value is an empirical value determined based on the parameter amount of the initial base large language model and the parameter amount of the sub-migration fine-tuning layer. For example, the corresponding relationship between the parameter amount of the initial base large language model, the parameter amount of the sub-migration fine-tuning layer, and the first preset value can be preset based on human experience, then, after the initial base large language model and the sub-migration fine-tuning layer are selected, the first preset value can be obtained based on the corresponding relationship.

[0118] In order to make the initial base large language model better understand the output of the above-mentioned initial migration fine-tuning layer, this embodiment can release the parameters of the preset layer in the initial base large language model, so as to train the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model based on the training text and the training text semantic vector, and obtain the post-training migration fine-tuning layer and the post-training base large language model.

[0119] It should be understood that the number of parameters in each layer of any large language model is very large. In order to improve the training efficiency as much as possible while achieving the purpose of accurately generating the subject text of the present application, the above-mentioned preset layers can be the last N layers, so that the present embodiment can quickly and accurately complete the generation of the subject text by using only a small proportion of the trainable parameters of the initial base large language model.

[0120] Optionally, the size of N mentioned above can be an empirical value; optionally, the size of N can also be determined according to a generation evaluation index, which refers to an index for evaluating the quality of the topic text output by a concatenated model consisting of a post-training transfer fine-tuning layer and a post-training base large language model.

[0121] Optionally, the generation evaluation indicator is the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) indicator, which is a common indicator for evaluating the quality of text generation. It measures the similarity between the generated text and the reference text by comparing the overlapping words or phrases between them.

[0122] More specifically, the generation evaluation index may be a ROUGE-1 index and / or a ROUGE-2 index, where the ROUGE-1 index considers the overlap rate of 1gram (i.e., a single word), and the ROUGE-2 index considers the overlap rate of 2grams (i.e., a phrase consisting of two consecutive words).

[0123] Optionally, the ROUGE-1 indicator may specifically be a recall rate (optionally, the recall rate may be the indicator that this embodiment mainly focuses on), a precision rate, etc.

[0124] Step S503, determine whether the topic text output by the concatenated model consisting of the trained transfer fine-tuning layer and the trained base large language model meets the text generation requirements, if so, execute step S504, if not, execute step S505.

[0125] Step S504: determine the splicing model as a text generation model.

[0126] Step S505: adjust the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer, and return to step S502.

[0127] In a possible implementation, the splicing model composed of the post-training transfer fine-tuning layer and the post-training base large language model can be directly used as the above-mentioned text generation model. However, through experimental research, it is found that after the first preset value is set based on artificial experience, sometimes the readability of the subject text output by the splicing model is poor. In order to solve this problem, this embodiment can first determine whether the subject text output by the above-mentioned splicing model meets the text generation requirements. If it does, the splicing model is determined as the text generation model. If it does not, the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is adjusted, and then retrained, that is, after adjusting the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer, the parameters of the new initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model are retrained.

[0128] Optionally, the above-mentioned process of "determining whether the topic text output by the splicing model composed of the post-training transfer fine-tuning layer and the post-training base large language model meets the text generation requirements" may include: obtaining the text semantic vector of the verification text as the verification text semantic vector, inputting the verification text semantic vector into the splicing model composed of the post-training transfer fine-tuning layer and the post-training base large language model, obtaining the topic text corresponding to the verification text semantic vector output by the splicing model, and finally determining whether the topic text corresponding to the verification text semantic vector meets the text generation requirements.

[0129] Here, the text generation requirements are pre-set requirements. Optionally, the text generation requirements may include quality requirements (such as readability requirements), accuracy requirements, etc., which are not specifically limited in this application.

[0130] Optionally, the text generation requirements can be determined based on the above-mentioned ROUGE indicator. For example, this embodiment can calculate the ROUGE value based on the subject text output by the splicing model and the standard subject text corresponding to the verification text semantic vector, and then compare it with a preset threshold. If the calculated ROUGE value is greater than or equal to the preset threshold, it is determined that the subject text corresponding to the verification text semantic vector meets the text generation requirements. If the calculated ROUGE value is less than the preset threshold, it is determined that the subject text corresponding to the verification text semantic vector does not meet the text generation requirements.

[0131] The above process of "adjusting the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer" includes: increasing the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer, or reducing the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer. For example, increasing the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer according to a preset step size, or reducing the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer, etc., which is not specifically limited in this application.

[0132] It should be noted that, when the above-mentioned generation evaluation indicator is the ROUGE indicator and the text generation requirements are determined based on the above-mentioned ROUGE indicator, the above-mentioned process of "adjusting the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer" may include: adjusting the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer and / or the size of N.

[0133] In summary, this embodiment involves the parameters of the preset layer in the initial base large language model in the training, which can ensure that the large language model after training can better understand the training multi-angle feature vector, thereby improving the readability and accuracy of the subject text.

[0134] When the topic text output by the splicing model does not meet the text generation requirements, this embodiment can adjust the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer, and then retrain the model. This method can ensure that the final generated text generation model outputs a topic text that meets the requirements in terms of accuracy and readability, thereby further improving the readability and accuracy of the topic text.

[0135] The following is an introduction to the process of "step S502, training the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model based on the training text and the training text semantic vector to obtain the post-training migration fine-tuning layer and the post-training base large language model".

[0136] See also Figure 8 , which is a schematic diagram of the process of initial migration fine-tuning layer and initial base large language model pre-training provided in this application.

[0137] like Figure 8The training text semantic vector shown is obtained by encoding the training text. The encoding process can refer to the process of "obtaining the text semantic vector of the training text" in the previous text, and will not be repeated here. Figure 8 As shown, the training text "I like dogs <end>"Get the semantic vector A of the training text, the training text "This is my pen <end>"Get the training text semantic vector B, training text"Good luck <end>"Get the training text semantic vector C.

[0138] In this embodiment, the training text semantic vector may be input into the initial migration fine-tuning layer to obtain the training multi-angle feature vector output by the initial migration fine-tuning layer.

[0139] As mentioned above, the initial transfer fine-tuning layer includes at least one sub-transfer fine-tuning layer. Each sub-transfer fine-tuning layer is used to map the training text semantic vector into a sub-vector. The sub-vectors mapped by all the sub-transfer fine-tuning layers are concatenated to obtain the above-mentioned training multi-angle feature vector.

[0140] For example, Figure 8 Take the initial transfer fine-tuning layer including 2 (i.e., hyperparameter prefix_lenth=2) sub-transfer fine-tuning layers as an example for introduction. The 2 sub-transfer fine-tuning layers can map the training text semantic vector A to sub-vector A0 and sub-vector A1 respectively, map the training text semantic vector B to sub-vector B0 and sub-vector B1 respectively, and map the training text semantic vector C to sub-vector C0 and sub-vector C1 respectively. Then, sub-vector A0 and sub-vector A1 are spliced ​​front and back to obtain the training multi-angle feature vector A, sub-vector B0 and sub-vector B1 are spliced ​​front and back to obtain the training multi-angle feature vector B, and sub-vector C0 and sub-vector C1 are spliced ​​front and back to obtain the training multi-angle feature vector C.

[0141] like Figure 8 In this embodiment, the training text can also be left-aligned, and a preset prompt word can be added to the head of the left-aligned training text. <promption>, add a terminator to the end of the left-aligned training text <end>, get the preprocessed training text.

[0142] The preset prompt words are used to enable the initial base large language model to understand the instruction action and make predictions based on the instruction action. <promption>It can be "Given the preceding context, please give three paragraphs, the first starting with Title:, second with Abstract:, third with Keywords:, each should be no less than 256 words:\n". Of course, the preset prompt words can also be set to other words according to actual needs, which is not limited in this application.

[0143] It should also be noted that after specifying a preset prompt word and obtaining a text generation model based on the previous training process, if the preset prompt word needs to be adjusted, it is necessary to adjust the preset prompt word and then train the parameters of the transfer fine-tuning layer in the text generation model (that is, the trained transfer fine-tuning layer of the previous text that constitutes the text generation model) and the parameters of the preset layer in the base large language model in the text generation model (that is, the trained base large language model of the previous text that constitutes the text generation model) again to adapt to the adjustment of the preset prompt word.

[0144] In order to facilitate computer understanding and processing, this embodiment can first use the tokenizer in the initial base large language model to process the pre-processed training text into an initial token sequence ( Figure 8 The initial word sequence is displayed in text form to facilitate viewing and understanding by those skilled in the art, the same below).

[0145] Here, "using the tokenizer in the initial base large language model to process the preprocessed training text into an initial token sequence" specifically means using the tokenizer in the initial base large language model to process each word contained in the preprocessed training text into a corresponding token, and obtaining an initial token sequence (i.e., an initial token sequence) corresponding to the preprocessed training text.

[0146] In this embodiment, the prediction output process of the initial base large language model is to predict the next character based on the previous character sequence, and then form a new previous character sequence based on the predicted character and the previous character sequence, and continue to predict the next character, and so on, until the end character is encountered. <end>Or generate a character sequence of a specified length. Given this autoregressive prediction method, during model training, the gold standard sequence needs to be one character ahead of the currently processed vector so that the initial base language model can correctly utilize contextual information.

[0147] Based on this, this embodiment can remove the last word in the initial word sequence and replace it with a blank character at the end. <pad>After completion, the first aligned word sequence is obtained, and then the encoder in the initial base large language model is used to process the first aligned word sequence into an embedding vector (embedding), and the embedding vector and the training multi-angle feature vector are concatenated front and back to obtain a concatenated vector (the concatenated vector is used as the "currently processed vector" mentioned above).

[0148] In addition, the present embodiment can also remove the first word in the initial word sequence and replace the word with a blank character at the end. <pad>The second aligned word sequence is obtained by padding, and then a second preset value of the target number is added to the head of the second aligned word sequence, and the blank characters in the second aligned word sequence are modified to the second preset value to obtain a gold standard sequence, wherein the target number is equal to the number of sub-transfer fine-tuning layers contained in the transfer fine-tuning layer.

[0149] Here, the second preset value refers to a minimum value. When the loss value is subsequently calculated, the loss value at the second preset value position is not included in the total loss value used to train the initial migration fine-tuning layer and the initial base large language model.

[0150] Optionally, the second preset value is "-100". It should be noted that "-100" represents the minimum value required by the initial base large language model and is not limited to a specific value.

[0151] Specifically, Figure 8 , the initial word-gram sequence is a left-aligned sequence. Due to the different lengths of different training texts, there is a gap at the end of the shorter initial word-gram sequence (such as the initial word-gram sequence corresponding to "I like dogs").

[0152] Because the input of the initial base language model needs to be a sequence of equal length, it is also necessary to end it with a blank character. <pad>Pad the length to get the first aligned word sequence.

[0153] The first aligned word-unit sequence mentioned above can be sent to the encoder in the initial base large language model for encoding to obtain the embedding vector corresponding to the first aligned word-unit sequence.

[0154] In this embodiment, the input vector format of the initial base large language model can be constructed as [text semantic vector after mapping transformation (i.e., training multi-angle feature vector)] + [promption] + [lable] format. As described above, "promption" is used to enable the initial base large language model to understand the instruction action, thereby generating a more accurate response, and "lable" is used to indicate the correct output of the input data. Based on this, this embodiment can splice the embedding vector and the previous training multi-angle feature vector to obtain a spliced ​​vector as the above-mentioned input vector.

[0155] In order to obtain the gold standard sequence that is one character ahead, this embodiment also needs to remove the first word in the initial word sequence, such as Figure 8 , the first word in the initial word sequence is <promption>Then we can <promption>The first word in is removed, and the remaining words are recorded as <promption-1>.

[0156] Similarly, because the input of the initial base language model needs to be a sequence of equal length, it is necessary to use blank characters on the basis of the above. <pad>The end length is padded to obtain the second aligned word sequence.

[0157] In addition, it is necessary to add a second preset value of the target number at the head of the second aligned word sequence, for example, Figure 8 Add two "-100" to the head of the second aligned word sequence, and replace the meaningless whitespace characters in the second aligned word sequence with <pad>Modify to the second preset value.

[0158] In order to indicate which characters need to be predicted during the initial fine-tuning of the migration layer and the initial base large language model training, it is also necessary to determine the mask sequence corresponding to the splicing vector, wherein the mask sequence is a sequence composed of mask values, the mask value at the position where the placeholder in the splicing vector is located is the first value, and the mask value at the non-placeholder position is the second value, and the placeholder includes a blank character and a terminator.

[0159] The second value indicates that the character at the non-placeholder position of the initial large language model needs to be predicted.

[0160] like Figure 8 , in the mask sequence, the whitespace character <pad>and end character <end>The mask value at the position is 0, and the mask value at the remaining positions is 1. It is worth noting that <promption>, sub-vector A0, sub-vector A1, sub-vector B0, sub-vector B1, sub-vector C0 and sub-vector C1 all contain multiple characters, so in the mask sequence, <promption>, sub-vector A0, sub-vector A1, sub-vector B0, sub-vector B1, sub-vector C0 and sub-vector C1 should contain the mask value "1" corresponding to multiple characters respectively, to distinguish them. Figure 8 Indicated by the shaded "1".

[0161] Finally, this embodiment can train the parameters of the initial transfer fine-tuning layer and the parameters of the preset layer based on the mask sequence, the concatenated vector and the gold standard sequence to obtain the trained transfer fine-tuning layer and the trained base large language model.

[0162] Optionally, the process of "training parameters of the initial migration fine-tuning layer and parameters of the preset layer based on the mask sequence, the concatenated vector and the gold standard sequence" may include: inputting the concatenated vector into the initial base large language model to obtain the training topic text output by the initial base large language model, calculating the total loss value Loss based on the training topic text, the mask sequence and the gold standard sequence (as described above, the loss value at the second preset value position is not included in the total loss value), releasing the parameters of the preset layer in the initial base large language model, and training the parameters of the initial migration fine-tuning layer and the parameters of the preset layer based on the calculated total loss value.

[0163] This embodiment provides a training process for all parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model. The total loss value is calculated for back propagation, and after multiple epochs (in model training, inputting all training data into the model once is called an epoch), the trained migration fine-tuning layer and the trained base large language model are obtained.

[0164] This embodiment can indicate the character positions that the initial base large language model needs to predict by setting a mask sequence; by setting a second preset value in the gold standard sequence, it can be indicated that when the initial base large language model calculates the total loss value, the loss value generated by the training multi-angle feature vector is not calculated, thereby ensuring that the total loss value is only related to the preset prompt word (which is essentially also text) and the training text, thereby improving the accuracy of the total loss value, and further improving the accuracy of the post-training transfer fine-tuning layer and the post-training base large language model.

[0165] A method for generating a topic text provided in an embodiment of the present application is introduced above, and a device for executing the above-mentioned method for generating a topic text will be introduced below.

[0166] See also Fig. 9 , Fig. 9 This is a schematic diagram of a structure of a device for generating a topic text provided in an embodiment of the present application. Fig. 9 As shown, the device may include:

[0167] Vector acquisition module 601, used to acquire the target cooperative topic vector to be converted into text;

[0168] A space mapping module 602, used to process the target cooperative topic vector into a target multi-angle feature vector in a target mapping space using a transfer fine-tuning layer in a pre-trained text generation model, wherein the target mapping space refers to a mapping space that can be processed by a base large language model in the text generation model;

[0169] The text prediction module 603 is used to use the base large language model in the text generation model to process the target multi-angle feature vector into a topic text corresponding to the target cooperative topic vector, wherein the parameters of the migration fine-tuning layer and the parameters of the preset layer in the base large language model are trained using the training text semantic vector annotated with the training text label as the training data, and the training text semantic vector and the target cooperative topic vector are in the same mapping space.

[0170] In a possible implementation, the training process of the text generation model in the above-mentioned space mapping module may include:

[0171] Obtaining a text semantic vector of the training text as the training text semantic vector;

[0172] Based on the training text and the training text semantic vector, the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model are trained to obtain the post-training migration fine-tuning layer and the post-training base large language model, wherein the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is a first preset value at the initial construction time;

[0173] Determine whether the topic text output by the concatenated model consisting of the post-training transfer fine-tuning layer and the post-training base large language model meets the text generation requirements;

[0174] If so, the concatenation model is determined as a text generation model;

[0175] If not, the number of sub-transfer fine-tuning layers included in the initial transfer fine-tuning layer is adjusted, and the parameters of the initial transfer fine-tuning layer trained based on the training text and the training text semantic vector and the parameters of the preset layer in the initial base large language model are returned.

[0176] In a possible implementation, the process of obtaining the text semantic vector of the training text in the above-mentioned spatial mapping module may include:

[0177] Extract the title, abstract and keywords of the training text;

[0178] Generate prompt sentences corresponding to the training text based on the title, abstract and keywords;

[0179] Align the prompt sentence corresponding to the training text to the left, and add a terminator to the end of the prompt sentence to obtain the processed prompt sentence corresponding to the training text;

[0180] The text semantic vector of the training text is obtained according to the processed prompt sentence.

[0181] In a possible implementation, the process of training the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model based on the training text and the training text semantic vector in the above-mentioned spatial mapping module may include:

[0182] Inputting the training text semantic vector into the initial transfer fine-tuning layer to obtain the training multi-angle feature vector output by the initial transfer fine-tuning layer, wherein each sub-transfer fine-tuning layer included in the initial transfer fine-tuning layer is used to map the training text semantic vector into a sub-vector, and the training multi-angle feature vector is obtained by concatenating the sub-vectors mapped by all the sub-transfer fine-tuning layers;

[0183] Align the training text to the left, and add a preset prompt word to the head of the left-aligned training text to obtain the preprocessed training text;

[0184] The preprocessed training text is processed into an initial word unit sequence using the word unit device in the initial base large language model;

[0185] Remove the last token in the initial token sequence and fill the end with a blank character to obtain the first aligned token sequence;

[0186] Process the first aligned word sequence into an embedding vector using the encoder in the initial base large language model;

[0187] Concatenate the embedding vector and the training multi-angle feature vector forward and backward to obtain a concatenated vector;

[0188] Remove the first token in the initial token sequence and fill the end with a blank character to obtain the second aligned token sequence;

[0189] Adding a second preset value of the target number to the head of the second aligned word sequence, and modifying the blank characters in the second aligned word sequence to the second preset value, to obtain a gold standard sequence, wherein the target number is equal to the number of sub-transfer fine-tuning layers included in the transfer fine-tuning layer;

[0190] Determine a mask sequence corresponding to the splicing vector, wherein the mask value at the position where the placeholder in the splicing vector is located is a first value, and the mask value at the non-placeholder position is a second value, and the placeholder includes a blank character and a terminator;

[0191] The parameters of the initial transfer fine-tuning layer and the parameters of the preset layer are trained based on the mask sequence, the concatenated vector and the gold standard sequence to obtain the post-training transfer fine-tuning layer and the post-training base large language model.

[0192] In a possible implementation, the process of training the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the spatial mapping module based on the mask sequence, the splicing vector and the gold standard sequence may include:

[0193] Input the concatenated vector into the initial base large language model to obtain the training topic text output by the initial base large language model;

[0194] Calculate the total loss value based on the training subject text, mask sequence and gold standard sequence;

[0195] Release the parameters of the preset layers in the initial base large language model;

[0196] The parameters of the initial transfer fine-tuning layer and the parameters of the preset layer are trained based on the total loss value.

[0197] In a possible implementation, the above-mentioned preset layers are the last N layers of the base large language model, and the size of N is determined according to the generation evaluation index.

[0198] In a possible implementation, the first preset value is an empirical value determined based on the parameter amount of the initial base large language model and the parameter amount of the sub-migration fine-tuning layer.

[0199] In a possible implementation, the sub-transfer fine-tuning layer uses a multi-layer linear perceptron or a Transformer encoder.

[0200] The subject text generation device provided in this application corresponds to the subject text generation method provided in the previous text. For details, please refer to the previous introduction and will not be repeated here.

[0201] The present application also provides an electronic device in an embodiment. Fig.10 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0202] like Fig.10 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0203] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0204] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any method for generating a subject text provided in the embodiment of the present application.

[0205] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any method for generating a subject text provided in an embodiment of the present application.

[0206] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0207] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0208] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0209] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.< / promption> < / promption> < / end> < / pad> < / pad> < / pad> < / promption> < / promption> < / pad> < / pad> < / pad> < / end> < / promption> < / end> < / promption> < / end> < / end> < / end>

Claims

1. A method for generating a topic text, characterized in that: include: Obtaining a target cooperative topic vector for text conversion; Using a transfer fine-tuning layer in a pre-trained text generation model, the target cooperative topic vector is processed into a target multi-angle feature vector in a target mapping space, wherein the target mapping space refers to a mapping space processable by a base large language model in the text generation model; The target multi-angle feature vector is processed into a topic text corresponding to the target cooperative topic vector by using the base large language model in the text generation model, wherein the parameters of the migration fine-tuning layer and the parameters of the preset layer in the base large language model are obtained by training with the training text semantic vector annotated with the training text label as the training data, and the training text semantic vector and the target cooperative topic vector are in the same mapping space; The training process of the text generation model includes: Obtaining a text semantic vector of the training text as the training text semantic vector; Based on the training text and the semantic vector of the training text, training parameters of the initial migration fine-tuning layer and parameters of the preset layer in the initial base large language model, obtaining a post-training migration fine-tuning layer and a post-training base large language model, wherein the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is a first preset value at the time of initial construction; Determine whether the topic text output by the concatenated model consisting of the post-training transfer fine-tuning layer and the post-training base large language model meets the text generation requirements; If yes, the concatenation model is determined as the text generation model; If not, the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is adjusted, and the parameters of the initial migration fine-tuning layer trained based on the training text and the training text semantic vector and the parameters of the preset layer in the initial base large language model are returned.

2. The method for generating a topic text according to claim 1, characterized in that: The obtaining of the text semantic vector of the training text includes: Extracting the title, abstract and keywords of the training text; Based on the title, the abstract and the keywords, generating a prompt sentence corresponding to the training text; Align the prompt sentence corresponding to the training text to the left, and add an end mark to the end of the prompt sentence to obtain a processed prompt sentence corresponding to the training text; A text semantic vector of the training text is obtained according to the processed prompt sentence.

3. The method for generating a topic text according to claim 1, characterized in that: The training of the parameters of the initial migration fine-tuning layer and the parameters of the preset layer in the initial base large language model based on the training text and the training text semantic vector includes: Inputting the training text semantic vector into the initial migration fine-tuning layer to obtain a training multi-angle feature vector output by the initial migration fine-tuning layer, wherein each of the sub-migration fine-tuning layers included in the initial migration fine-tuning layer is used to map the training text semantic vector into a sub-vector, and the training multi-angle feature vector is obtained by concatenating the sub-vectors mapped by all the sub-migration fine-tuning layers; Left-aligning the training text, and adding a preset prompt word to the head of the left-aligned training text to obtain a pre-processed training text; Processing the preprocessed training text into an initial word unit sequence using a word unit device in the initial base large language model; The last word in the initial word sequence is removed and the end is padded with a blank character to obtain a first aligned word sequence; Processing the first aligned word-unit sequence into an embedding vector using an encoder in the initial base large language model; Concatenate the embedding vector and the training multi-angle feature vector forward and backward to obtain a concatenated vector; The first word in the initial word-unit sequence is removed, and the end is padded with the blank character to obtain a second aligned word-unit sequence; Adding a second preset value of the target number to the head of the second aligned word sequence, and modifying the blank characters in the second aligned word sequence to the second preset value, to obtain a gold standard sequence, wherein the target number is equal to the number of sub-transfer fine-tuning layers included in the transfer fine-tuning layer; Determine a mask sequence corresponding to the splicing vector, wherein the mask value at the position where the placeholder in the splicing vector is located is a first value, and the mask value at the non-placeholder position is a second value, and the placeholder includes the blank character and the end character; The parameters of the initial transfer fine-tuning layer and the parameters of the preset layer are trained based on the mask sequence, the concatenated vector and the gold standard sequence to obtain the trained transfer fine-tuning layer and the trained base large language model.

4. The method for generating a topic text according to claim 3, characterized in that: The step of training the parameters of the initial migration fine-tuning layer and the parameters of the preset layer based on the mask sequence, the splicing vector and the gold standard sequence includes: Inputting the concatenated vector into the initial base large language model to obtain the training topic text output by the initial base large language model; Calculating a total loss value based on the training subject text, the mask sequence, and the gold standard sequence; Release the parameters of the preset layer in the initial base large language model; Parameters of the initial migration fine-tuning layer and parameters of the preset layer are trained based on the total loss value.

5. The method for generating a topic text according to claim 1, characterized in that: The preset layers are the last N layers of the base large language model, and the size of N is determined according to the generation evaluation index.

6. The method for generating a topic text according to claim 1, characterized in that: The first preset value is an empirical value determined based on the parameter amount of the initial base large language model and the parameter amount of the sub-migration fine-tuning layer.

7. The method for generating a topic text according to claim 1, characterized in that: The sub-migration fine-tuning layer adopts a multi-layer linear perceptron or a Transformer encoder.

8. A device for generating a topic text, characterized in that: include: A vector acquisition module, used to acquire a target cooperative topic vector to be converted into text; A space mapping module, used to process the target cooperative topic vector into a target multi-angle feature vector under a target mapping space using a transfer fine-tuning layer in a pre-trained text generation model, wherein the target mapping space refers to a mapping space processable by a base large language model in the text generation model; A text prediction module, used to process the target multi-angle feature vector into a topic text corresponding to the target cooperative topic vector by using the base large language model in the text generation model, wherein the parameters of the migration fine-tuning layer and the parameters of the preset layer in the base large language model are obtained by training with the training text semantic vector annotated with the training text label as training data, and the training text semantic vector and the target cooperative topic vector are in the same mapping space; The training process of the text generation model includes: Obtaining a text semantic vector of the training text as the training text semantic vector; Based on the training text and the semantic vector of the training text, training parameters of the initial migration fine-tuning layer and parameters of the preset layer in the initial base large language model, obtaining a post-training migration fine-tuning layer and a post-training base large language model, wherein the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is a first preset value at the time of initial construction; Determine whether the topic text output by the concatenated model consisting of the post-training transfer fine-tuning layer and the post-training base large language model meets the text generation requirements; If yes, the concatenation model is determined as the text generation model; If not, the number of sub-migration fine-tuning layers included in the initial migration fine-tuning layer is adjusted, and the parameters of the initial migration fine-tuning layer trained based on the training text and the training text semantic vector and the parameters of the preset layer in the initial base large language model are returned.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the method for generating a subject text as described in any one of claims 1 to 7.

Images