A method and device for text generation

Abstract

The present application provides a method and device for text generation, which are used to improve the text quality and text richness on the basis of ensuring the generality of the text generation method, get rid of the dependence on manual labor and improve the efficiency. Receive indication information for indicating text prediction, where the indication information is sent by the terminal device after receiving a voice command for playing a multimedia resource; input the structured data corresponding to the multimedia resource into a trained text generation model, perform multiple rounds of text sequence prediction to obtain a target text, and return it to the terminal device; the structured data includes text information for characterizing the attributes of the multimedia resource and text information related to the content of the multimedia resource; the process of each round of text sequence prediction is as follows: according to the text information in the text sequence predicted in the previous round, determine the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set; predict the text sequence of the current round according to the first probability and the second probability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and provides a method and device for text generation. Background Art

[0002] With the development of technology, speech technology is widely used in human-computer interaction. The terminal device receives the user's voice command and returns a response message to the user.

[0003] Currently, the text information corresponding to the response message is generated based on pre-written traditional rules, and the generation method is fixed and not natural and vivid enough.

[0004] In order to make the text information more natural and vivid, the traditional rules are still used to determine ordinary multimedia resources, while special multimedia resources are generated by manual writing to generate the corresponding text information. Due to the continuous increase of special multimedia resources, it is necessary to continuously write manually for the newly added special multimedia resources. The manual writing cost is high and the efficiency is low, and the number of resources that can be covered is limited. Summary of the Invention

[0005] The embodiments of this application provide a method and device for message feedback to ensure the generality of the text generation method, improve the quality and richness of the text, get rid of the dependence on manual work and improve the efficiency.

[0006] In a first aspect, the embodiments of this application provide a method for text generation, and the method includes:

[0007] Receiving indication information for indicating text prediction, where the indication information is sent after the terminal device receives the voice command triggered by the user to play the multimedia resource;

[0008] Inputting the structured data corresponding to the multimedia resource into the trained text generation model, and performing multiple rounds of text sequence prediction through the text generation model to obtain the target text, and returning the target text to the terminal device; wherein, the structured data includes text information for characterizing the attributes of the multimedia resource and text information related to the content of the multimedia resource, and the text information includes words and / or sentences;

[0009] Wherein, the process of each round of text sequence prediction is:

[0010] Determining the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round;

[0011] Predicting the text sequence of the current round according to the first probability and the second probability.

[0012] In a possible implementation manner, determining a first probability of each piece of text information in the structured data according to the text information in the text sequence predicted in the previous round includes:

[0013] For the text information not included in the text sequence predicted in the previous round in the structured data, using the probability in the probability distribution of the attention mechanism and the first weight of the current round to obtain the first probability of this piece of text information in the structured data; or

[0014] For the text information included in the text sequence predicted in the previous round in the structured data, reducing the probability of the text information predicted in the previous round in the probability distribution of the attention mechanism, and obtaining the first probability of this piece of text information in the structured data according to the reduced probability and the first weight of the current round;

[0015] Determining a second probability of each piece of text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round includes:

[0016] Determining the distribution probability of the high-frequency word set by using the context vector obtained from the probability of the attention mechanism in the current round and the state sequence decoded according to the text information in the text sequence predicted in the previous round;

[0017] According to the distribution probability and the second weight of the current round, obtaining the second probability of each piece of text information in the high-frequency word set.

[0018] In a possible implementation manner, determining the first weight and the second weight of the current round through the following method:

[0019] Determining the context vector of the structured data according to the probability obtained by the attention mechanism in the current round;

[0020] Determining the second weight of the current round according to the context vector of the structured data and the state sequence decoded according to the text information in the text sequence predicted in the previous round;

[0021] Determining the first weight of the current round according to the second weight.

[0022] In a possible implementation manner, predicting the text sequence of the current round according to the first probability and the second probability includes:

[0023] For the text information with both the first probability and the second probability, adding the first probability and the second probability to obtain the final probability of the text information; and for the text information with only the first probability, using the first probability as the final probability of the text information; and for the text information with only the second probability, using the second probability as the final probability of the text information;

[0024] Predict the text information of the current round from all text information according to the final probability of the text information;

[0025] Combine the text information of the current round with the text sequence predicted in the previous round to determine the candidate text sequence of the current round, where the text sequence predicted in the previous round includes the text information of the previous round used to predict the text information of the current round;

[0026] Predict the text sequence of the current round according to the probability value of the candidate text sequence.

[0027] In a possible implementation, predicting the text sequence of the current round according to the probability value of the candidate text sequence includes:

[0028] For the last round of prediction, use the text sequence with the largest probability value in the candidate text sequences of the current round as the text sequence of the current round; or

[0029] For non-last-round predictions, according to the sorting rule of the candidate text sequences from largest to smallest probability value, select the first N candidate text sequence probabilities from the candidate text sequences as the text sequence of the current round, where N is a positive integer.

[0030] In a possible implementation, determine the text information related to the multimedia resource content through the following methods:

[0031] Determine the text information related to the multimedia resource content according to the number of occurrences of sentences in the content information corresponding to the multimedia resource; and / or

[0032] Determine the text information related to the multimedia resource content according to the sentiment score of the sentences in the content appreciation information corresponding to the multimedia resource.

[0033] In a second aspect, an embodiment of the present application provides a text generation device, which includes: a memory and a processor;

[0034] The memory is used to store program code;

[0035] The processor is configured to read the program code in the memory and execute receiving instruction information for indicating text prediction, where the instruction information is sent after the terminal device receives a voice instruction triggered by the user to play the multimedia resource;

[0036] Input the structured data corresponding to the multimedia resource into the trained text generation model, perform multi-round text sequence prediction through the text generation model to obtain the target text, and return the target text to the terminal device; where the structured data includes text information for characterizing the attributes of the multimedia resource and text information related to the multimedia resource content, and the text information includes words and / or sentences;

[0037] Among them, the process of predicting the text sequence in each round is as follows:

[0038] According to the text information in the text sequence predicted in the previous round, determine the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set;

[0039] Predict the text sequence of the current round according to the first probability and the second probability.

[0040] In a third aspect, an embodiment of the present application provides a text generation device, which includes: a receiving module, a prediction module, and a return module, where:

[0041] The receiving module is configured to receive indication information for indicating text prediction, where the indication information is sent after the terminal device receives a voice instruction triggered by the user to play a multimedia resource;

[0042] The prediction module is configured to input the structured data corresponding to the multimedia resource into a trained text generation model, and perform multi-round text sequence prediction through the text generation model to obtain a target text;

[0043] The return module is configured to return the target text to the terminal device and feed it back to the user through the terminal device;

[0044] Among them, the structured data contains text information for characterizing the attributes of the multimedia resource and text information related to the content of the multimedia resource, and the text information includes words and / or sentences;

[0045] Among them, the process of the prediction module performing each round of text sequence prediction is as follows:

[0046] According to the text information in the text sequence predicted in the previous round, determine the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set;

[0047] Predict the text sequence of the current round according to the first probability and the second probability.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method for message feedback provided by the embodiment of the present application is implemented.

[0049] The beneficial effects of the embodiments of the present application are as follows:

[0050] An embodiment of the present application provides a method and device for text generation. The method includes receiving text prediction indication information, which is sent by a terminal device after receiving a voice command triggered by a user to play a multimedia resource; inputting structured data corresponding to the multimedia resource into a trained text generation model, and performing multi-round text sequence prediction through the text generation model to obtain a target text. The structured data corresponding to the multimedia resource includes text information for characterizing the attributes of the multimedia resource and text information related to the content of the multimedia resource, and the text information is a word and / or a sentence.

[0051] The target text is determined based on the structured data corresponding to the multimedia resource and the trained text generation model. Inputting the structured data corresponding to multiple multimedia resources into the text generation model can obtain corresponding target texts. Therefore, the method of outputting the target text through the text generation model is applicable to all multimedia resources, ensuring the generality of the text generation method. Moreover, the structured data contains text information for characterizing the attributes of the multimedia resource and high-quality text information related to the content of the multimedia resource, and the text information includes words and / or sentences. Therefore, the target text generated based on the structured data contains information related to the multimedia resource, making the quality of the target text higher. In addition, the method of using the text generation model to generate the target text supports flexible sentence patterns and expressions, making the expression more natural and vivid, improving the richness of the text. And the target text can positively introduce the multimedia resource to enhance the user experience. It no longer adopts traditional rules written manually in advance and manually written specific responses, reducing the manual workload, getting rid of the dependence on manual work and improving the efficiency.

[0052] Other features and advantages of the present application will be described in the following specification, and some of them will become obvious from the specification, or can be understood by implementing the present application. The objectives and other advantages of the present application can be realized and obtained through the structures specifically pointed out in the written specification, claims, and drawings. Brief Description of the Drawings

[0053] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 It is a schematic diagram of an application scenario for text generation provided by an embodiment of the present application;

[0055] Figure 2 It is a schematic diagram of a terminal device provided by an embodiment of the present application;

[0056] Figure 3 A flowchart of a text generation method provided by an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a trained text generation model provided by an embodiment of the present application;

[0058] Figure 5 A schematic diagram of a Beam search decoding method provided by an embodiment of the present application;

[0059] Figure 6 A flowchart of an overall text generation method provided by an embodiment of the present application;

[0060] Figure 7 A structural diagram of a text generation device provided by an embodiment of the present application;

[0061] Figure 8 A structural diagram of a text generation apparatus provided by an embodiment of the present application. Detailed implementation manners

[0062] In order to make the objectives, technical solutions and beneficial effects of the present application clearer and more understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above accompanying drawings are used to distinguish similar data / information, and do not necessarily need to describe a specific order or sequence. It should be understood that such data / information used may be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0064] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated information, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated information before and after.

[0065] It should be noted that the text generation implementation manner of the present application is applied to a scenario where a user interacts with a terminal device by voice. After receiving the user's voice command, the terminal device returns a corresponding response message to the user, and the text information corresponding to the response message is generated by the server and sent to the terminal device.

[0066] Figure 1 A schematic diagram of an application scenario for text generation is exemplarily shown. As Figure 1 shown, this application scenario includes a terminal device 100 and a server 101.

[0067] The terminal device 100 and the server 101 communicate data through various communication methods. Here, the terminal device 100 is allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 101 can provide various contents and interactions to the terminal device 100. Exemplarily, the terminal device 100 can send and receive information, for example: receive a voice command triggered by a user, send the data in the voice command to the server 101 through the network, and receive text information sent by the server 101, etc.

[0068] The server 101 can be a group or multiple groups, and can be one type or multiple types of servers.

[0069] The terminal device 100 can be a device with voice interaction function such as a mobile phone, a computer, a television, a speaker, etc.

[0070] Figure 2 A hardware configuration block diagram of the terminal device 100 is exemplarily shown. As Figure 2 shown, the terminal device 100 may include a controller 210, a tuner demodulator 220, a communication interface 230, a detector 240, an input / output interface 250, a video processor 260-1, an audio processor 260-2, a display 280, an audio output 270, a memory 290, a power supply, and an infrared receiver.

[0071] The display 280 is a component for receiving an image signal input from the video processor 260-1 and displaying video content, images, and a menu control interface. The display 280 includes a display screen component for presenting a picture and a driving component for driving image display. The displayed video content can come from broadcast television content, that is to say, various broadcast signals received through wired or wireless communication protocols. Or, it can display various image contents received from a network server through a network communication protocol.

[0072] At the same time, the display 280 also displays a user control UI interface generated in the terminal device 100 and used to control the terminal device 100.

[0073] And, depending on the type of the display 280, it also includes a driving component for driving the display. Or, if the display 280 is a projection display, it may further include a projection device and a projection screen.

[0074] The communication interface 230 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communication interface 230 can be a Wifi chip 231, a Bluetooth communication protocol chip 232, a wired Ethernet communication protocol chip 233, or other network communication protocol chips or near-field communication protocol chips, as well as an infrared receiver (not shown in the figure).

[0075] The terminal device 100 can establish the sending and receiving of control signals and data signals with an external control device or content providing device through the communication interface 230. In addition, the infrared receiver is an interface for receiving infrared control signals of the control device 100 (such as an infrared remote control, etc.).

[0076] The detector 240 is a component for the terminal device 100 to collect signals from the external environment or interact with the outside. The detector 240 includes a light receiver 242, a sensor for collecting the intensity of ambient light, and can adaptively display parameter changes by collecting ambient light, etc.

[0077] And it includes an image collector 241, such as a camera, a webcam, etc., which can be used to collect external environmental scenes, as well as to collect user attributes or user interaction gestures, can adaptively change display parameters, and can also recognize user gestures to achieve the function of interacting with users.

[0078] In some other exemplary embodiments, the detector 240 can also be a temperature sensor, etc. For example, by sensing the ambient temperature, the terminal device 100 can adaptively adjust the display color temperature of the image. For example, in an environment with a relatively high temperature, the terminal device 100 can adjust the display color temperature of the image to be cooler, or in an environment with a relatively low temperature, the terminal device 100 can adjust the display image to be warmer.

[0079] In some other exemplary embodiments, the detector 240 can also be a sound collector, etc., such as a microphone, which can be used to receive the user's voice, including voice signals of the user's control instructions for the terminal device 100, or to collect ambient sounds for identifying the type of ambient scene, and the terminal device 100 can adapt to ambient noise.

[0080] The input / output interface 250 controls the data transmission between the terminal device 100 and other external devices under the control of the controller 210. For example, it receives data such as video signals, audio signals, or command instructions from external devices.

[0081] Among them, the input / output interface 250 can include, but is not limited to, the following: any one or more interfaces such as a high-definition multimedia interface HDMI interface 251, an analog or data high-definition component input interface 253, a composite video input interface 252, a USB input interface 254, an RGB port (not shown in the figure), etc.

[0082] In some other exemplary embodiments, the input / output interface 250 may also form a composite input / output interface with the above-mentioned multiple interfaces.

[0083] The tuner demodulator 220 receives broadcast television signals through wired or wireless reception, and can perform modulation and demodulation processes such as amplification, mixing, and resonance, demodulating the television audio and video signals and EPG data signals carried in the user-selected television channel frequency from multiple wireless or wired broadcast television signals.

[0084] The tuner demodulator 220 can, according to user selection and under the control of the controller 210, respond to the television signal frequency selected by the user and the television signal carried by that frequency.

[0085] There are many ways for the tuner demodulator 220 to receive signals. Depending on the broadcast standard of the television signal, they include: terrestrial broadcast, cable broadcast, satellite broadcast, or Internet broadcast signals, etc.; and depending on the modulation type, it can be digital modulation or analog modulation. Depending on the type of received television signal, it can be analog signal and digital signal.

[0086] In some other exemplary embodiments, the tuner demodulator 220 can also be in an external device, such as an external set-top box, etc. In this way, the set-top box outputs television audio and video signals after modulation and demodulation, and inputs them into the terminal device 100 through the input / output interface 250.

[0087] The video processor 260-1 is used to receive external video signals and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal, and can obtain signals that can be directly displayed or played on the terminal device 100.

[0088] Exemplarily, the video processor 260-1 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc.

[0089] Among them, the demultiplexing module is used to demultiplex the input audio and video data stream. For example, if the input is MPEG-2, the demultiplexing module demultiplexes it into video signals and audio signals, etc.

[0090] The video decoding module is used to process the demultiplexed video signals, including decoding and scaling processes, etc.

[0091] The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signals generated by the graphics generator according to user input or generated by itself with the scaled video image to generate an image signal for display.

[0092] The frame rate conversion module is used to convert the frame rate of the input video, such as converting a 60Hz frame rate to a 120Hz frame rate or a 240Hz frame rate. Usually, the format is implemented by means of frame interpolation.

[0093] The display formatting module is used to receive the video output signal after frame rate conversion and change the signal to conform to the display format signal, such as outputting an RGB data signal.

[0094] The audio processor 260-2 is used to receive external audio signals, perform decompression, decoding, noise reduction, digital-to-analog conversion, amplification processing, etc. according to the standard codec protocol of the input signal, and obtain a sound signal that can be played on the speaker.

[0095] In some other exemplary embodiments, the video processor 260-1 may be composed of one or more chips. The audio processor 260-2 may also be composed of one or more chips.

[0096] Moreover, in some other exemplary embodiments, the video processor 260-1 and the audio processor 260-2 may be separate chips or integrated with the controller 210 in one or more chips.

[0097] The audio output 270 receives the sound signal output by the audio processor 260-2 under the control of the controller 210, such as the speaker 272, and in addition to the speaker 272 carried by the terminal device 100 itself, it can output to the external audio output terminal 274 of the sound generating device of the external device, such as an external audio interface or a headphone interface, etc.

[0098] The power supply, under the control of the controller 210, provides power supply support for the terminal device 100 with the power input from the external power supply. The power supply may include a built-in power circuit installed inside the terminal device 100, or it may be an external power supply installed outside the terminal device 100, and a power interface for providing an external power supply in the terminal device 100.

[0099] The user input interface is used to receive the user's input signal and then send the received user input signal to the controller 210. The user input signal may be a remote control signal received through an infrared receiver or various user control signals received through a network communication module.

[0100] Exemplarily, the user control signal is a voice signal. The user inputs a user command by voice. The user input interface in the terminal device 100 sends the voice signal to the controller 210, and further responds to the user's input through the controller 210.

[0101] In some embodiments, if a user enters a user command through a graphical user interface (GUI) displayed on the display 280, the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, if the user enters a user command by inputting a specific sound or gesture, the user input interface receives the user input command by identifying the sound or gesture through a sensor.

[0102] The controller 210 controls the operation of the terminal device 100 and responds to user operations through various software control programs stored in the memory 290.

[0103] As Figure 2 shown, the controller 210 includes a RAM 213, a ROM 214, a graphics processor 216, a CPU processor 212, a communication interface 218, such as: a first interface 218-1 to an nth interface 218-n, and a communication bus. Among them, the RAM 213, the ROM 214, the graphics processor 216, the CPU processor 212, and the communication interface 218 are connected through the bus.

[0104] The RAM 213 is used to store instructions for various system startups. For example, when a power-on signal is received, the power supply of the terminal device 100 starts to boot, and the CPU processor 212 runs the system startup instructions in the ROM, copies the operating system stored in the memory 290 to the RAM 213, so as to start running the startup operating system. After the operating system starts up, the CPU processor 212 then copies various application programs in the memory 290 to the RAM 213, and then starts to run and start various application programs.

[0105] The graphics processor 216 is used to generate various graphic objects, such as: icons, operation menus, and graphic displays of user input instructions, etc. It includes an arithmetic unit that performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes. And it includes a renderer that generates various objects obtained based on the arithmetic unit and displays the rendered results on the display 280.

[0106] The CPU processor 212 is used to execute the operating system and application program instructions stored in the memory 290. And according to various interactive instructions received from the outside, it executes various application programs, data, and content, so as to finally display and play various audio and video contents.

[0107] In some exemplary embodiments, the CPU processor 212 may include multiple processors. The multiple processors may include a main processor and multiple or one sub-processor. The main processor is used to perform some operations of the terminal device 100 in the pre-power-on mode and / or display screen operations in the normal mode. The multiple or one sub-processor is used to perform an operation in a standby mode or other states.

[0108] The controller 210 can control the overall operation of the terminal device 100. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 280, the controller 210 can perform operations related to the object selected by the user command.

[0109] Among them, the object can be any one of the selectable objects, such as a hyperlink or an icon. Operations related to the selected object, such as: operations for displaying a page, document, image, etc. connected to the hyperlink, or operations for executing a program corresponding to the icon. The user command for selecting a UI object can be a command input through various input devices (such as a mouse, keyboard, touchpad, etc.) connected to the terminal device 100 or a voice command corresponding to the user's spoken voice.

[0110] The memory 290 includes various software modules stored for driving the terminal device 100. For example, various software modules stored in the memory 290 include: a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules, etc.

[0111] Among them, the basic module is a bottom-layer software module used for signal communication between various hardware in the postpartum care terminal device 100 and for sending processing and control signals to upper-layer modules. The detection module is a management module used for collecting various information from various sensors or user input interfaces, performing analog-to-digital conversion, and analyzing and managing.

[0112] For example: The voice recognition module includes a voice parsing module and a voice instruction database module. The display control module is a module used for controlling the display 280 to display image content, and can be used for playing multimedia image content and UI interface and other information. The communication module is a module used for performing control and data communication with external devices. The browser module is a module used for performing data communication between browsing servers. The service module is a module used for providing various services and various application programs.

[0113] At the same time, the memory 290 is also used to store received external data and user data, images of various items in various user interfaces, and visual effect diagrams of focus objects, etc.

[0114] In related technologies, after the terminal device receives a voice command triggered by a user, it replies to the user with a response message according to the target text; among them, the target text is determined based on pre-written traditional rules or based on manually written specific replies.

[0115] The following takes the music playback scenario as an example for illustration:

[0116] The user-triggered voice command is a request to play a certain song. After the terminal device receives the song playback request, it replies with a corresponding response message to the playback request.

[0117] If the requested song is a non-specific song, a pre-written traditional rule is used to determine the target text corresponding to the response message.

[0118] If the requested song is a specific song, a manually written specific reply is used to determine the target text corresponding to the response message.

[0119] As shown in Table 1, it is an example of determining the target text based on the pre-written traditional rule in the related art.

[0120] Table 1

[0121]

[0122] It can be seen from Table 1 that regardless of which song the user requests to play, the target text corresponding to the response message feedback by the terminal device is determined according to the pre-written traditional rule; obviously, the supported sentence patterns are single, new statements cannot be generated, and the generated text is not natural and vivid enough.

[0123] As shown in Table 2, it is an example of determining the target text by using a manually written specific reply for specific songs in the related art. Among them, the specific song can be a song on the soaring chart, the new song chart or the hot song chart.

[0124] Table 2

[0125] The user's voice command Terminal device A Terminal device B I want to listen to ABC , Happy every day, now playing ABC for you. Okay, let's enjoy Zhang San's ABC together. I want to listen to BYY Okay, here's a BYY for you. For you, Zhou Zhou's BYY. I want to listen to HCY Okay, here comes a different HCY. For you, Li Si's HYC.

[0126] It can be seen from Table 2 that using the manually written specific reply to generate the target text corresponding to the specific song improves the richness of the text. However, this method is not applicable to all songs. If this method is used to determine the target text for all songs, specific replies need to be written for each song, resulting in a large amount of manual work and high costs.

[0127] It should be noted that the music playback scenario is only an exemplary scenario, and the technical solution of the present application can also be applied to any scenario of interaction by voice, such as video playback, audio playback, poetry recitation, story reading, article learning, etc.

[0128] In view of this, the embodiments of the present application provide a method and device for text generation to ensure the generality of the text generation method while improving the richness of the text, saving the manual operation and maintenance cost, getting rid of the dependence on manual work and improving the efficiency.

[0129] In this application, after the terminal device receives a voice command for playing a multimedia resource triggered by a user, it sends indication information for indicating text prediction to the server; correspondingly, after the server receives the indication information for indicating text prediction, it inputs the structured data corresponding to the multimedia resource into a trained text generation model, performs multiple rounds of text sequence prediction through the text generation model to obtain a target text, and returns the obtained target text to the terminal device, so that the terminal device feeds back a response message to the user in a voice manner according to the target text. The structured data corresponding to the multimedia resource includes text information for characterizing the attributes of the multimedia resource and text information related to the content of the multimedia resource, and the text information is words and / or sentences.

[0130] The target text corresponding to the response message is determined based on the structured data corresponding to the multimedia resource and the trained text generation model. Inputting the structured data corresponding to multiple multimedia resources into the text generation model can obtain the corresponding target text. Therefore, the method of outputting the target text through the text generation model is applicable to all multimedia resources, ensuring the generality of the text generation method; and the structured data contains text information for characterizing the attributes of the multimedia resource and high-quality text information related to the content of the multimedia resource, and the text information includes words and / or sentences. Therefore, the target text generated based on the structured data contains information related to the multimedia resource, making the target text of higher quality; and the method of using the text generation model to generate the target text supports flexible and variable sentence patterns and expressions, making the expression more natural and vivid, improving the richness of the text, and the target text can introduce the multimedia resource positively to enhance the user experience; it no longer uses traditional rules and specific responses pre-written manually, getting rid of the dependence on manual work, reducing the manual workload, and improving the efficiency. And the target text generated based on the structured data corresponding to the multimedia resource can make a simple and positive introduction and guidance to the multimedia resource, stimulating the user's audiovisual interest and enhancing the user experience.

[0131] Next, in combination with the above-described application scenario, the method for message feedback provided by an exemplary embodiment of the present application will be described with reference to the accompanying drawings. It should be noted that the above application scenario is only shown for the convenience of understanding the spirit and principle of the present application, and the embodiments of the present application are not limited in this regard.

[0132] As Figure 3 shown, a method for text generation provided by an embodiment of the present application includes the following steps:

[0133] Step 300, receiving indication information for indicating text prediction.

[0134] Among them, the indication information is sent by the terminal device after receiving a voice command for playing a multimedia resource triggered by a user.

[0135] In the present application, the terminal device receives a voice command for playing a multimedia resource triggered by a user, sends indication information for indicating text prediction to the server, so as to receive a target text generated by the server according to the indication information, and feeds back a response message to the user in a voice manner according to the target text.

[0136] Therefore, after receiving the indication information sent by the terminal device, the server will perform text prediction, such as step 301.

[0137] Step 301: Input the structured data corresponding to the multimedia resource into a trained text generation model, perform multiple rounds of text sequence prediction through the text generation model to obtain a target text, and return the target text to the terminal device.

[0138] When performing text prediction, it is first necessary to determine the structured data corresponding to the multimedia resource.

[0139] In the present application, the structured data corresponding to the multimedia resource is determined in the following manner:

[0140] Step 1: Obtain the target information corresponding to the multimedia resource, where the target information includes but is not limited to: attribute information, content information, content appreciation information.

[0141] Specifically, the target information corresponding to the multimedia resource can be obtained from the network through web crawler technology; obtaining the target information from the network through web crawler technology is only an example, and any technology that can obtain the target information corresponding to the multimedia resource is applicable to the present application.

[0142] The attribute information is used to represent the name, author, etc. of the multimedia resource, the content information is used to represent the content of the multimedia resource, and the content appreciation information is used to represent information such as the appreciation and analysis of the content of the multimedia resource.

[0143] Taking the multimedia resource as a song as an example:

[0144] The attribute information includes Chinese name, foreign name, lyricist, composer, arranger, singer, release time, album, music style, singing duration, etc.;

[0145] The content information is the lyric content;

[0146] The content appreciation information is the appreciation content corresponding to the song.

[0147] Step 2: Determine the text information related to the content of the multimedia resource from the determined content information and / or content appreciation information.

[0148] When determining the text information related to the content of the multimedia resource in the content information, it is determined according to the number of occurrences of sentences in the content information corresponding to the multimedia resource;

[0149] For example, the sentence with the most occurrences in the content information is used as the text information related to the content of the multimedia resource.

[0150] Taking the multimedia resource as a song as an example, the "highlight sentence" in the lyrics is used as the text information related to the content of the multimedia resource. The structural forms of songs are mostly "two-part form, three-part form", and the "highlight sentence of the song" appears repeatedly. Therefore, the whole sentence with the most occurrences in the lyrics is the "highlight sentence", and the "highlight sentence" is the text information related to the content of the multimedia resource in this application.

[0151] When determining the text information related to the content of the multimedia resource in the content appreciation information, since the content appreciation information exists in paragraphs and there are no repeated sentences, the content appreciation information is cut into multiple sentences, and the text information related to the content of the multimedia resource is determined according to the emotional scores of the sentences in the content appreciation information.

[0152] When cutting the content appreciation information into multiple sentences, if the sentence is too long, it will lead to too long text information related to the content of the multimedia resource, causing storage pressure, and resulting in too long voice playback time, consuming the user's patience and reducing the user experience; if the sentence is too short, there may be a possibility of incomplete semantics, and at this time, the semantics of the text information related to the content of the multimedia resource may be incomplete, failing to play a positive introduction role.

[0153] Therefore, according to the preset rules, the content appreciation information is cut into multiple sentences. The preset rules are to cut according to punctuation marks, or / and cut according to the context semantics to ensure the integrity of the cut sentences;

[0154] For example, after cutting the sentences according to punctuation, if there is an associative relationship or a connecting relationship between two adjacent sentences and they are connected by conjunctions, prepositions, or pronouns, and the sum of the lengths of the two adjacent sentences is less than the preset threshold, then the two adjacent sentences are taken as one sentence; if the sum of the lengths exceeds the preset threshold, then the conjunctions and prepositions in the sentence are deleted. Or

[0155] If the sentence is too short, then multiple consecutive sentences are combined into one sentence, and the sentence length is controlled by the threshold.

[0156] In a possible implementation, in order to ensure the integrity of the cut sentences, before cutting the content appreciation information, the pronouns in the content appreciation information can also be replaced with the nouns they refer to according to the context information of the content appreciation information.

[0157] In this application, when determining the text information related to the multimedia resource content according to the sentiment score of the sentences in the content appreciation information, the segmented sentences can be input into the sentiment scoring model, so that the sentiment scoring model outputs the sentiment scores of each sentence. According to the sorting rule of the sentiment scores from high to low, at least one sentence with a high score is selected from the multiple segmented sentences as the text information related to the target media resource content.

[0158] For example, the sentence with the highest sentiment score is selected as the text information related to the multimedia resource content, or the top four sentences in terms of sentiment score are selected as the text information related to the multimedia resource content. A high sentiment rating is ensured to guarantee that the text information is positive.

[0159] It should be noted that when determining the text information related to the multimedia resource content from the content appreciation information, short, semantically complete, and independently high-quality sentences should be selected as the text information related to the multimedia resource content.

[0160] Step 3: Determine the structured data of the multimedia resource according to the determined attribute information and the text information related to the multimedia resource content.

[0161] The determined structured data contains the text information for characterizing the attributes of the multimedia resource and the text information related to the multimedia resource content. The determined structured data is input into the trained text generation model, and the trained text generation model performs multiple rounds of text sequence prediction to obtain the target text.

[0162] As Figure 4 shown, in the schematic diagram of a trained text generation model provided by an embodiment of this application, the trained text generation model includes an encoder (encoding), a decoder (decoding), an attention mechanism, a copy mechanism, and a coverage mechanism.

[0163] The encoder can adopt an RNN (or its variant LSTM / GRU). For example, a single-layer bidirectional LSTM (Long Short Term Memory Network) is adopted. In the encoder encoding stage, the input is the structured data, and the output of the encoder encoding stage is the encoded hidden layer state sequence h i , where the hidden layer state sequence is the vector corresponding to the structured data.

[0164] The decoder can adopt an RNN (or its variant LSTM / GRU). For example, a single-layer unidirectional LSTM is adopted. In the decoder decoding stage, the input is the text sequence determined in the previous round, and the output is a decoded state sequence s t, for predicting the text sequence of the current round.

[0165] The Attention mechanism is used to determine the probability of structured data, so as to determine which text information in the structured data is more important in the current round according to the probability of the structured data determined by the Attention mechanism during the process of predicting the text sequence in each round.

[0166] During the process of predicting the text sequence, the text information in the text sequence is determined according to the text information in the high-frequency word set and the text information in the structured data, where the high-frequency word set is the candidate word list of the text generation model.

[0167] When predicting the text information of the text sequence, the entire high-frequency word set is faced, that is, text information is selected from the high-frequency word set. The dimension of the weight matrix in the model is related to the size of the high-frequency word set. The larger the high-frequency word set, the larger the dimension of the weight matrix in the model, the computing pressure of the model increases, and the computing efficiency decreases. Therefore, the size of the high-frequency word set affects the computing efficiency of the model.

[0168] Generally, the size of the high-frequency word set needs to reach the order of 100,000, that is, during the process of predicting the text sequence in each round, there are 100,000 choices for the text information of the current round, resulting in a large computing pressure and low computing efficiency of the model. In this application, in order to improve the computing efficiency, a preset number of high-frequency words are stored in the high-frequency word set, and the preset number is less than 100,000. Preferably, 30,000 high-frequency words are stored in the high-frequency word set.

[0169] Since the text information used to characterize the attributes of the multimedia resource corresponding to different multimedia resources is different from the text information related to the content of the multimedia resource, and each multimedia resource corresponds to multiple text information related to the content, the amount of structured data is huge due to the large number of text information related to the content. In order to prevent the operation pressure on the text generation model, the structured data cannot be stored in the high-frequency word set. Therefore, a copy mechanism is introduced in this application to copy text information from the input structured data.

[0170] Because during the process of predicting the text sequence, the text information in the structured data appears repeatedly, which affects the smoothness of the sentence, a coverage mechanism is introduced to prevent the same text information from appearing repeatedly.

[0171] The following combines Figure 4 , and details the process of predicting the text sequence in each round of the embodiment of this application:

[0172] When predicting the text sequence in each round: according to the text information in the text sequence predicted in the previous round, determine the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set;

[0173] Predict the text sequence of the current round according to the first probability and the second probability.

[0174] In a possible implementation manner, when determining the first probability of each text information in the structured data according to the text information in the text sequence predicted in the previous round:

[0175] For the text information not included in the text sequence predicted in the previous round in the structured data, use the probability in the attention mechanism probability distribution and the first weight of the current round to obtain the first probability of this text information in the structured data; or

[0176] For the text information included in the text sequence predicted in the previous round in the structured data, reduce the probability of the text information predicted in the previous round in the attention mechanism probability distribution, and obtain the first probability of this text information in the structured data according to the reduced probability and the first weight of the current round.

[0177] As Figure 4 shown, input the structured data into the trained text generation model, and encode the structured data through the encoder of the trained text generation model to obtain the state sequence of the encoded output corresponding to the structured data.

[0178] Input the state sequence of the encoded output and the state sequence corresponding to the text information included in the text sequence predicted in the previous round into the attention mechanism. After determining that the structured data does not contain the text information predicted in the previous round, use the attention mechanism to obtain the probability of the current round of the structured data; or after determining that the structured data contains the text information predicted in the previous round, use the attention mechanism to reduce the probability of the text information in the structured data that is the same as the text information predicted in the previous round, and determine the probability of the current round of the structured data; as Figure 4 shown, the probability of the structured data is represented in a bar form, and the higher the height of the bar, the more important the text information is in predicting the current text sequence.

[0179] In this application, because the coverage mechanism is introduced, the probability of the text information in the structured data that has been previously focused on by the attention mechanism can be reduced. The coverage mechanism is mainly used to maintain a coverage vector: where a t′ is the attention distribution, c t is the coverage vector, and t is defined over all previously decoded time steps.

[0180] This coverage vector is the cumulative sum of the probability distributions of the structured data calculated in all previous rounds during the text prediction process, and records the text information in the structured data that has been included in the text sequence.

[0181] Use this coverage vector as an additional input to the attention mechanism, which affects the probability of the structured data in the current round in the attention mechanism.

[0182] The purpose of doing this is that when the trained text generation model determines the probability of structured data during text generation, it can determine the text information in the structured data already included in the text sequence, avoiding the repeated occurrence of the same text information in a certain text sequence.

[0183] Meanwhile, the coverage mechanism also includes a coverage loss to penalize the repeatedly determined text information. The coverage loss is as follows:

[0184] It should be noted that this coverage loss only penalizes the text information in the repeatedly occurring structured data, and does not force the text generation model to pay attention to every text information in the structured data.

[0185] In this application, the probability of the structured data obtained by using the attention mechanism is determined through the following formula:

[0186]

[0187] a t = softmax(e t )

[0188] where s t represents the decoding state, h i represents the encoded hidden state, and W h , b attn and v T are parameters.

[0189] It should be noted that when determining the first probability of each text information in the structured data, not only the probability of the structured data obtained by using the attention mechanism needs to be determined, but also the first weight of the current round needs to be determined.

[0190] In this application, the first weight of the current round is determined through the following method:

[0191] Determine the context vector of the structured data according to the probability obtained by the attention mechanism in the current round;

[0192] Determine the second weight of the current round according to the context vector of the structured data and the state sequence decoded and output according to the text information in the text sequence predicted in the previous round;

[0193] Determine the first weight of the current round according to the second weight.

[0194] As Figure 4 shown, after determining the probability obtained by the attention mechanism in the current round, perform weighted summation processing to obtain the context vector of the structured data. As Figure 4 shown in, the Context Vector represents the dynamic representation of the structured data obtained after weighted summation, which is called the context vector. The context vector is:

[0195]

[0196] Determine the second weight p according to the context vector and the state sequence of the decoded output gen :

[0197]

[0198] where x t represents the decoded input, that is, the vector representation of the text information predicted in the previous round; and b ptr are parameters.

[0199] The first weight is: 1 - p gen .

[0200] Therefore, use the probability of the structured data obtained by the attention mechanism and the first weight to determine the first probability of each text information in the structured data of the current round.

[0201] When determining the second probability of each text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round:

[0202] Use the context vector obtained by the probability of the attention mechanism in the current round, and the state sequence decoded according to the text information in the text sequence predicted in the previous round to determine the distribution probability of the high-frequency word set;

[0203] According to the distribution probability and the second weight of the current round, obtain the second probability of each text information in the high-frequency word set.

[0204] As Figure 4 shown, the distribution probability in the high-frequency word set is represented in a bar form, and the higher the height of the bar, the greater the probability distribution:

[0205] The distribution probability of the high-frequency word set is:

[0206] where V′, V, b, b′ are parameters.

[0207] It should be noted that when determining the second probability of each text information in the high-frequency word set, not only the distribution probability in the high-frequency word set needs to be determined, but also the second weight of the current round needs to be determined.

[0208] In this application, the second weight is determined in the following manner:

[0209] Based on the probability obtained from the attention mechanism of the current round, determine the context vector of the structured data;

[0210] Based on the context vector of the structured data and the state sequence decoded and output from the text information in the text sequence predicted in the previous round, determine the second weight of the current round.

[0211] Specifically, reference can be made to the determination method of the first weight, which will not be elaborated here.

[0212] In this application, when predicting the text sequence of the current round based on the first probability and the second probability:

[0213] For the text information with both the first probability and the second probability, add the first probability and the second probability to obtain the final probability of the text information; and for the text information with only the first probability, use the first probability as the final probability of the text information; and for the text information with only the second probability, use the second probability as the final probability of the text information;

[0214] As Figure 4 shown, add the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set to obtain the final probability.

[0215] In this application, the final probability is determined by the following formula:

[0216]

[0217] w represents the predicted text information.

[0218] Based on the final probability of the text information, predict the text information of the current round from all the text information;

[0219] According to the rule of ranking the final probabilities from large to small, select multiple text information with probability rankings greater than the preset value from all the text information as the text information of the current round.

[0220] Combine the text information of the current round with the text sequence predicted in the previous round to determine the candidate text sequence of the current round;

[0221] Based on the probability value of the candidate text sequence, predict the text sequence of the current round.

[0222] In a possible implementation manner, predicting the text sequence of the current round according to the probability values of the candidate text sequences includes:

[0223] For the last-round prediction, use the text sequence with the largest probability value among the candidate text sequences in the current round as the text sequence of the current round; or

[0224] For non-last-round predictions, according to the sorting rule of the candidate text sequences from largest to smallest probability values, select the first N candidate text sequence probabilities from the candidate text sequences as the text sequence of the current round, where N is a positive integer.

[0225] In this application, during the text sequence prediction process, the Beam search decoding method is used to predict the text sequence of the current round. As Figure 5 shown, a Beam search decoding method provided by an embodiment of this application includes:

[0226] Step 1, in the first round of text sequence prediction, determine the first text information in the target text, and this text information is the text sequence generated in the current round;

[0227] Suppose there are three candidate text information, namely A1, A2, and A3, where the sequence probability of A1 is -1.05, the sequence probability of A2 is -0.92, and the sequence probability of A3 is -1.39. At this time, select the two text information with the largest probabilities, namely A1 and A2. Then the text sequence of the current round is A1, A2;

[0228] Step 2, according to any text information determined in the previous round, respectively determine the candidate text information in the current round, and combine the determined candidate text information with the text sequence containing the text information of the previous round used to determine the candidate text information to determine the candidate text sequence of the current round;

[0229] For example, for A1 determined in the previous round, determine the text information in the current round as B1 and B2. The text sequences in the current round are: A1B1, A1B2;

[0230] For A2 determined in the previous round, determine the text information in the current round as B1, B2, and B3. The text sequences in the current round are: A2B1, A2B2, A2B3;

[0231] The candidate text sequences in the current round are: A1B1, A1B2, A2B1, A2B2, A2B3.

[0232] Step 3, select the text sequence of the current round from the candidate text sequences of the current round according to the sequence probability;

[0233] For example, the sequence probabilities of each candidate text sequence are -2.95, -1.27, -1.84, -1.61, and -3.22 respectively;

[0234] Take the two candidate text sequences with the highest sequence probabilities in the candidate text sequences as the text sequences for the current round. The text sequences for the current round are: A1B2 and A2B2.

[0235] Continuously repeat the processes of step 2 and step 3 until an end symbol is encountered or the text sequence reaches the preset length.

[0236] When an end symbol is encountered or the text sequence reaches the preset length, determine that the current round is the last round of prediction. At this time, take the text sequence with the highest candidate text sequence probability value in the current round as the text sequence for the current round, and use the text sequence for the current round as the target text and output it.

[0237] Assume that A1B2 and A2B2 encounter an end symbol in the next step. Then select A1B2 with the highest sequence probability among A1B2 and A2B2 as the target text output by the trained text generation model.

[0238] In this application, compared with splicing after locally determining text information, the Beam search decoding method can better ensure the quality and smoothness of text information.

[0239] It should be noted that the text generation model adopted in this application is a trained text generation model. When training the text generation model, the input of the model is the structured data of the text information used to represent the attributes of the multimedia resource and the text information related to the content of the multimedia resource, and the output of the model is the target text manually annotated based on the structured data. In order to make the text information related to the content of the multimedia resource output by the text generation model be able to blend with other generated words and ensure the smoothness and logic of the finally generated target text, before model training, it is necessary to design the position of the text information related to the content of the multimedia resource in the model training statement. Since the text information related to the content of the multimedia resource is a complete short sentence for semantic expression, it is therefore segmented from the context by punctuation marks in the training statement and made into a half sentence, and through model training, learn the mapping relationship between the text information in the input structured data and its position in the output text, so that the text information related to the content of the multimedia resource will not appear alone between two words, ensuring the smoothness and logic of the text information.

[0240] Exemplary:

[0241] In this application, the structured data input into the trained text generation model is:

[0242] {Song name: QHC

[0243] Singer: Zhou Zhou

[0244] Lyricist: Fang Fang

[0245] Composer: Zhou Zhou

[0246] Music Style: Pop / Chinese Style

[0247] Release Date: 2007

[0248] Sentence 1: It's as if a Chinese landscape painting is right in front of you and can be slowly appreciated.

[0249] Sentence 2: It's quite different from the music styles of Zhou Zhou's previous songs.

[0250] Sentence 3: TQSDYY,EWZDN}

[0251] Model Output:

[0252] Text 1: The lyrics of "QHC" are written by Fang Fang and the music is composed by Zhou Zhou. Let me accompany you to listen.

[0253] Text 2: "TQSDYY,EWZDN", presenting you with the song "QHC".

[0254] Text 3: Listening to this Chinese-style song "QHC", it's as if a Chinese landscape painting is right in front of you and can be slowly appreciated.

[0255] Text 4: This song "QHC" is quite different from the music styles of Zhou Zhou's previous songs. Let's enjoy it together.

[0256] In a possible implementation, the server pre-determines the target text corresponding to at least one multimedia resource through a trained text generation model, and stores the multimedia resource and its corresponding target text.

[0257] After receiving the voice command from the user to play the multimedia resource, the terminal device sends a request message to the server to request the target text corresponding to the response message;

[0258] Correspondingly, after receiving the request message, the server first checks whether the multimedia resource carried in the request message exists in the corresponding stored multimedia resource and target text;

[0259] If it exists, directly return the target text corresponding to the multimedia resource to the terminal device; or

[0260] If it doesn't exist, determine the target text corresponding to the multimedia resource through the text generation method in the above embodiment, return it to the terminal device, and update the corresponding stored multimedia resource and target text in the server for future use to reduce the computing pressure.

[0261] It should be noted that one multimedia resource corresponds to at least one target text, and each target text is generated by a corresponding trained text generation model; when selecting a target text to return to the terminal device, any one of the target texts is randomly selected.

[0262] When storing the multimedia resource and the target text information correspondingly in the server, it can be stored in the form of a database, such as in the form of an ES index. In this application, the stored multimedia resource and target text information are illustrated in the form of a table, as shown in Table 3:

[0263] Table 3

[0264]

[0265] Among them, "TQSDYY,EWZDN" is the climax sentence in the song "QHC", and "as if the landscape painting is right in front of you and can be slowly appreciated" and "quite different from the previous song styles of Zhou Zhou" are the sentences with relatively high emotional scores in the appreciation information of the song "QHC"; "BYGXLMGDF,NMLQNMBL" is the climax sentence in the song "BYG", and "showing the delicacy of men in love", "laying the unique love song style of Zhang San", and "Zhang San is singing affectionately, and the audience can also feel a touch of warmth from this" are the sentences with relatively high emotional scores in the appreciation information of "BYG". The text information contains rich and high-quality information about the song, which can simply and positively introduce the song to the user to improve the user's audio-visual interest.

[0266] As Figure 6 shown, an overall method for text generation provided by an embodiment of this application includes the following steps:

[0267] Step 600, the terminal device receives a voice command for playing a multimedia resource triggered by the user;

[0268] Step 601, the terminal device sends indication information for indicating text prediction to the server;

[0269] Step 602, the server receives the indication information for indicating text prediction and determines the structured data corresponding to the multimedia resource;

[0270] Step 603, the server inputs the structured data corresponding to the multimedia resource into the trained text generation model, and performs multiple rounds of text sequence prediction through the text generation model to obtain the target text;

[0271] Step 604, the server returns the target text to the terminal device;

[0272] Step 605, the terminal device feeds back a response message to the user in voice according to the target text.

[0273] Based on the same inventive concept, an embodiment of the present invention further provides a text generation device. Since this device corresponds to the text generation method in the embodiment of the present invention, and the principle of solving problems by this device is similar to that of the method, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be elaborated here.

[0274] As Figure 7 shown, it is a structural diagram of a text generation device provided by an embodiment of the present application. The device includes: a memory 700 and a processor 701. The memory 700 stores program codes. When the program codes are executed by the processor 701, the processor 701 is specifically configured to:

[0275] Receive indication information for indicating text prediction, where the indication information is sent by the terminal device after receiving a voice instruction for playing a multimedia resource triggered by the user;

[0276] Input the structured data corresponding to the multimedia resource into the trained text generation model, perform multiple rounds of text sequence prediction through the text generation model to obtain the target text, and return the target text to the terminal device; wherein, the structured data contains text information for characterizing the attributes of the multimedia resource and text information related to the content of the multimedia resource, and the text information includes words and / or sentences;

[0277] Among them, the process of each round of text sequence prediction is:

[0278] According to the text information in the text sequence predicted in the previous round, determine the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set;

[0279] Predict the text sequence of the current round according to the first probability and the second probability.

[0280] In a possible implementation manner, the processor 701 determines the first probability of each text information in the structured data according to the text information in the text sequence predicted in the previous round, including:

[0281] For the text information not included in the text sequence predicted in the previous round in the structured data, use the probability in the attention mechanism probability distribution and the first weight of the current round to obtain the first probability of the text information in the structured data; and / or

[0282] For the text information included in the text sequence predicted in the previous round in the structured data, reduce the probability of the text information predicted in the previous round in the attention mechanism probability distribution, and obtain the first probability of the text information in the structured data according to the reduced probability and the first weight of the current round.

[0283] In a possible implementation, the processor 701 determines the second probability of each piece of text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round, including:

[0284] Determining the distribution probability of the high-frequency word set by using the context vector obtained from the probability obtained by the attention mechanism in the current round and the state sequence decoded according to the text information in the text sequence predicted in the previous round;

[0285] According to the distribution probability and the second weight in the current round, obtain the second probability of each piece of text information in the high-frequency word set.

[0286] In a possible implementation, the processor 701 determines the first weight and the second weight in the current round in the following manner:

[0287] Determine the context vector of the structured data according to the probability obtained by the attention mechanism in the current round;

[0288] According to the context vector of the structured data and the state sequence decoded according to the text information in the text sequence predicted in the previous round, determine the second weight in the current round;

[0289] Determine the first weight in the current round according to the second weight.

[0290] In a possible implementation, the processor 701 predicts the text sequence in the current round according to the first probability and the second probability, including:

[0291] For the text information with the first probability and the second probability, add the first probability and the second probability to obtain the final probability of the text information; for the text information with only the first probability, use the first probability as the final probability of the text information; and for the text information with only the second probability, use the second probability as the final probability of the text information;

[0292] Predict the text information in the current round from all the text information according to the final probability of the text information;

[0293] Combine the text information in the current round with the text sequence predicted in the previous round to determine the candidate text sequence in the current round, where the text sequence predicted in the previous round includes the text information in the previous round used to predict the text information in the current round;

[0294] Predict the text sequence in the current round according to the probability value of the candidate text sequence.

[0295] In a possible implementation, the processor 701 predicts the text sequence in the current round according to the probability value of the candidate text sequence, including:

[0296] For the last round of prediction, use the text sequence with the highest probability value among the candidate text sequences in the current round as the text sequence for the current round; or

[0297] For non-last-round predictions, according to the sorting rule of arranging candidate text sequences in descending order of probability values, select the first N candidate text sequence probabilities from the candidate text sequences as the text sequence for the current round, where N is a positive integer.

[0298] In a possible implementation, the processor 701 determines the text information related to the multimedia resource content in the following manner:

[0299] Determine the text information related to the multimedia resource content according to the number of occurrences of sentences in the content information corresponding to the multimedia resource; and / or

[0300] Determine the text information related to the multimedia resource content according to the sentiment score of the sentences in the content appreciation information corresponding to the multimedia resource.

[0301] As Figure 8 shown, it is a structural diagram of a text generation device provided by an embodiment of the present application. The device includes: a receiving module 800, a prediction module 801, and a return module 802, where:

[0302] The receiving module 800 is configured to receive indication information for indicating text prediction, where the indication information is sent after the terminal device receives a voice instruction triggered by the user to play a multimedia resource;

[0303] The prediction module 801 is configured to input the structured data corresponding to the multimedia resource into a trained text generation model, and perform multiple rounds of text sequence prediction through the text generation model to obtain a target text;

[0304] The return module 802 is configured to return the target text to the terminal device;

[0305] Among them, the structured data contains text information for characterizing the attributes of the multimedia resource and text information related to the multimedia resource content, and the text information includes words and / or sentences;

[0306] Among them, the process of the prediction module 801 performing each round of text sequence prediction is:

[0307] According to the text information in the text sequence predicted in the previous round, determine the first probability of each text information in the structured data and the second probability of each text information in the high-frequency word set;

[0308] Predict the text sequence for the current round according to the first probability and the second probability.

[0309] In a possible implementation, the prediction module 801 determines the first probability of each piece of text information in the structured data according to the text information in the text sequence predicted in the previous round, including:

[0310] For the text information not included in the text sequence predicted in the previous round in the structured data, the first probability of this text information in the structured data is obtained by using the probability in the attention mechanism probability distribution and the first weight of the current round; and / or

[0311] For the text information included in the text sequence predicted in the previous round in the structured data, the probability of the text information predicted in the previous round in the attention mechanism probability distribution is reduced, and the first probability of this text information in the structured data is obtained according to the reduced probability and the first weight of the current round.

[0312] In a possible implementation, the prediction module 801 determines the second probability of each piece of text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round, including:

[0313] Determine the distribution probability of the high-frequency word set by using the context vector obtained from the probability obtained by the attention mechanism of the current round and the state sequence decoded according to the text information in the text sequence predicted in the previous round;

[0314] According to the distribution probability and the second weight of the current round, the second probability of each piece of text information in the high-frequency word set is obtained.

[0315] In a possible implementation, the prediction module 801 determines the first weight and the second weight of the current round in the following manner:

[0316] Determine the context vector of the structured data according to the probability obtained by the attention mechanism of the current round;

[0317] Determine the second weight of the current round according to the context vector of the structured data and the state sequence decoded according to the text information in the text sequence predicted in the previous round;

[0318] Determine the first weight of the current round according to the second weight.

[0319] In a possible implementation, the prediction module 801 predicts the text sequence of the current round according to the first probability and the second probability, including:

[0320] For the text information with both the first probability and the second probability, add the first probability and the second probability to obtain the final probability of the text information; for the text information with only the first probability, use the first probability as the final probability of the text information; and for the text information with only the second probability, use the second probability as the final probability of the text information;

[0321] Predict the text information of the current round from all text information according to the final probability of the text information;

[0322] Combine the text information of the current round with the text sequence predicted in the previous round to determine the candidate text sequence of the current round, where the text sequence predicted in the previous round includes the text information of the previous round used to predict the text information of the current round;

[0323] Predict the text sequence of the current round according to the probability value of the candidate text sequence.

[0324] In a possible implementation manner, the prediction module 801 predicts the text sequence of the current round according to the probability value of the candidate text sequence, including:

[0325] For the last round of prediction, use the text sequence with the largest probability value in the candidate text sequence of the current round as the text sequence of the current round; or

[0326] For non-last-round prediction, according to the sorting rule of the candidate text sequences from largest to smallest probability value, select the first N candidate text sequence probabilities from the candidate text sequences as the text sequence of the current round, where N is a positive integer.

[0327] In a possible implementation manner, the prediction module 801 determines the text information related to the multimedia resource content in the following manner:

[0328] Determine the text information related to the multimedia resource content according to the number of times the sentences appear in the content information corresponding to the multimedia resource; and / or

[0329] Determine the text information related to the multimedia resource content according to the sentiment score of the sentences in the content appreciation information corresponding to the multimedia resource.

[0330] In a possible implementation manner, the embodiments of the present application further provide a computer-storable medium, on which a computer program is stored, and when the program is executed by a processor, the method steps of text generation in the embodiments of the present application are implemented.

[0331] In a possible implementation manner, each aspect of the text generation method provided by the present application can also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to cause the computer device to execute the steps in the text generation method according to various exemplary embodiments of the present application described above in this specification.

[0332] The program product may employ any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0333] Those skilled in the art will appreciate that the embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0334] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0335] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0336] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide means for implementing the functions specified in the flowFigure 1 one process or multiple processes and / or blocks Figure 1 steps of the functions specified in one block or multiple blocks.

[0337] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A method for text generation, characterized in that, The method includes: Receiving indication information for indicating text prediction, where the indication information is sent after the terminal device receives a voice command triggered by the user to play a multimedia resource; Inputting the structured data corresponding to the multimedia resource into a trained text generation model, performing multiple rounds of text sequence prediction through the text generation model to obtain a target text, and returning the target text to the terminal device; wherein, the structured data contains text information for characterizing the attributes of the multimedia resource and text information related to the content of the multimedia resource, and the text information includes words and / or sentences; Wherein, the process of each round of text sequence prediction is: Determining a first probability of each text information in the structured data and a second probability of each text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round; Predicting the text sequence of the current round according to the first probability and the second probability; The determining the first probability of each text information in the structured data according to the text information in the text sequence predicted in the previous round includes: For the text information not included in the text sequence predicted in the previous round in the structured data, obtaining the first probability of the non-included text information in the structured data by using the probability of the structured data determined by the attention mechanism and the first weight of the current round; the attention mechanism is preset in the text generation model; and / or For the text information included in the text sequence predicted in the previous round in the structured data, using the attention mechanism to reduce the probability of the text information in the structured data that is the same as the text information predicted in the previous round, and obtaining the first probability of the text information in the structured data that is the same as the text information predicted in the previous round according to the reduced probability and the first weight of the current round; Determining the second probability of each text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round includes: After determining the probability of the structured data by using the attention mechanism of the current round, obtaining the context vector of the structured data through weighted summation processing; Determining the distribution probability of the high-frequency word set through the context vector and the state sequence decoded according to the text information in the text sequence predicted in the previous round; Obtaining the second probability of each text information in the high-frequency word set according to the distribution probability and the second weight of the current round.

2. The method according to claim 1, characterized in that, Determining the first weight and the second weight of the current round in the following manner: Determining the context vector of the structured data according to the probability obtained by the attention mechanism of the current round; Determining the second weight of the current round according to the context vector of the structured data and the state sequence decoded according to the text information in the text sequence predicted in the previous round; Determining the first weight of the current round according to the second weight.

3. The method according to claim 1, characterized in that The predicting the text sequence of the current round according to the first probability and the second probability includes: For the text information with the first probability and the second probability, add the first probability and the second probability to obtain the final probability of the text information; and for the text information with only the first probability, use the first probability as the final probability of the text information; and for the text information with only the second probability, use the second probability as the final probability of the text information; Predict the text information of the current round from all text information according to the final probability of the text information; Combine the text information of the current round with the text sequence predicted in the previous round to determine the candidate text sequence of the current round, where the text sequence predicted in the previous round includes the text information of the previous round used to predict the text information of the current round; Predict the text sequence of the current round according to the probability value of the candidate text sequence.

4. The method according to claim 3, characterized in that, The predicting the text sequence of the current round according to the probability value of the candidate text sequence includes: For the last round of prediction, use the text sequence with the largest probability value in the candidate text sequences of the current round as the text sequence of the current round; or For non-last-round prediction, according to the arrangement rule of the candidate text sequences from the largest to the smallest probability value, select the first N candidate text sequence probabilities from the candidate text sequences as the text sequence of the current round, where N is a positive integer.

5. The method according to any one of claims 1 to 4, characterized in that Determine the text information related to the multimedia resource content in the following manner: Determine the text information related to the multimedia resource content according to the number of times the sentences appear in the content information corresponding to the multimedia resource; and / or Determine the text information related to the multimedia resource content according to the sentiment score of the sentences in the content appreciation information corresponding to the multimedia resource.

6. A text generation device, characterized in that, The device includes: a memory and a processor; The memory is used to store program codes; The processor is used to read the program codes in the memory and execute receiving the indication information for text prediction, where the indication information is sent after the terminal device receives the voice instruction triggered by the user to play the multimedia resource; Input the structured data corresponding to the multimedia resource into the trained text generation model, perform multiple rounds of text sequence prediction through the text generation model to obtain the target text, and return the target text to the terminal device; where the structured data includes the text information used to characterize the attributes of the multimedia resource and the text information related to the multimedia resource content, and the text information includes words and / or sentences; Wherein, the process of each round of text sequence prediction is: According to the text information in the text sequence predicted in the previous round, determine the first probability of each text information in the structured data, and the second probability of each text information in the high-frequency word set; Predict the text sequence of the current round according to the first probability and the second probability; The processor determines the first probability of each text information in the structured data according to the text information in the text sequence predicted in the previous round, including: For the text information not included in the text sequence predicted in the previous round of the structured data, the first probability of the text information not included in the structured data is obtained by using the probability of the structured data determined by the attention mechanism and the first weight of the current round; the attention mechanism is preset in the text generation model; and / or For the text information included in the text sequence predicted in the previous round of the structured data, the probability of the text information in the structured data that is the same as the text information predicted in the previous round is reduced by using the attention mechanism, and the first probability of the text information in the structured data that is the same as the text information predicted in the previous round is obtained according to the reduced probability and the first weight of the current round; The processor determines the second probability of each text information in the high-frequency word set according to the text information in the text sequence predicted in the previous round, including: After using the attention mechanism of the current round to determine the probability of the structured data, the context vector of the structured data is obtained through weighted summation processing; The distribution probability of the high-frequency word set is determined through the context vector and the state sequence decoded and output according to the text information in the text sequence predicted in the previous round; According to the distribution probability and the second weight of the current round, the second probability of each text information in the high-frequency word set is obtained.

7. The device according to claim 6, characterized in that, The processor determines the first weight and the second weight of the current round in the following manner: According to the probability obtained by the attention mechanism of the current round, the context vector of the structured data is determined; According to the context vector of the structured data and the state sequence decoded and output according to the text information in the text sequence predicted in the previous round, the second weight of the current round is determined; According to the second weight, the first weight of the current round is determined.

8. The device according to claim 6, characterized in that, The processor predicts the text sequence of the current round according to the first probability and the second probability, including: For the text information with the first probability and the second probability, the first probability and the second probability are added to obtain the final probability of the text information; and for the text information with only the first probability, the first probability is used as the final probability of the text information; and for the text information with only the second probability, the second probability is used as the final probability of the text information; According to the final probability of the text information, the text information of the current round is predicted from all text information; The text information of the current round is combined with the text sequence predicted in the previous round to determine the candidate text sequence of the current round; According to the probability value of the candidate text sequence, the text sequence of the current round is predicted; Among them, the processor predicts the text sequence of the current round according to the probability value of the candidate text sequence, including: For the last round of prediction, the text sequence with the largest probability value among the candidate text sequences in the current round is used as the text sequence of the current round; or For non-final-round predictions, according to the rule of arranging the candidate text sequences in descending order of probability values, select the probabilities of the first N candidate text sequences from the candidate text sequences as the text sequences for the current round, where N is a positive integer.

Images