Text generation method and apparatus

Abstract

A text generation method and device are disclosed. Source data related to an object is acquired; one or more source data related to the object is acquired; the source data is analyzed to obtain one or more material information; the material information is parsed to obtain corresponding one or more text paragraph information; and the text paragraph information is used to generate a text describing the object. Thus, the user can directly and comprehensively understand the object according to the generated text without performing a large number of retrieval operations.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a text generation method and device. Background Art

[0002] With the rapid growth of information, object descriptions are characterized by multiple modalities and large amounts of information. Therefore, how to better understand objects without requiring users to manually perform extensive search operations is an urgent problem that needs to be solved.

[0003] Taking commodities as an example, the introduction information of commodities includes multimodal information such as text, pictures, and videos. Users need to browse the above-mentioned multiple modal information to fully understand the commodities, which will undoubtedly increase the user's browsing time and reduce the user experience.

[0004] Therefore, a solution is needed that enables users to better understand objects without having to perform a large number of search operations. Summary of the Invention

[0005] The purpose of the present disclosure is to provide a solution that enables users to better understand objects without having to perform a large number of search operations.

[0006] According to a first aspect of the present disclosure, a text generation method is provided for generating text related to an object, comprising: acquiring one or more source data related to the object; analyzing the source data to obtain one or more material information; parsing the material information to obtain one or more corresponding text paragraph information; and using the text paragraph information to generate text describing the object.

[0007] According to a second aspect of the present disclosure, a text generation device is provided for generating text related to an object, including: an acquisition module for acquiring one or more source data related to the object; an analysis module for analyzing the source data to obtain one or more material information; a parsing module for parsing the material information to obtain corresponding one or more text paragraph information; and a generation module for generating text describing the object using the text paragraph information.

[0008] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in the first aspect above.

[0009] According to a fourth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0010] The present disclosure obtains one or more source data related to an object, analyzes the source data to obtain one or more material information, parses the material information to obtain one or more corresponding text paragraph information, and uses the text paragraph information to generate text describing the object, so that users can directly understand the object in an all-round way based on the generated text without having to manually perform a large number of search operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0012] Figure 1 A schematic flowchart of a text generation method according to an embodiment of the present disclosure is shown.

[0013] Figure 2 A schematic flowchart of a text generation method according to another embodiment of the present disclosure is shown.

[0014] Figure 3 A schematic diagram of a product details page according to an embodiment of the present disclosure is shown.

[0015] Figure 4 A schematic diagram of the script construction principle according to an embodiment of the present disclosure is shown.

[0016] Figure 5 A schematic flowchart of constructing a script based on multimodal data of an object according to an embodiment of the present disclosure is shown.

[0017] Figure 6 A schematic diagram showing pictures of multiple products.

[0018] Figure 7 A schematic diagram of the content reorganization principle according to an embodiment of the present disclosure is shown.

[0019] Figure 8 A schematic diagram of the construction process of a broadcast framework according to an embodiment of the present disclosure is shown.

[0020] Figure 9 A schematic structural diagram of a text generation device according to an embodiment of the present disclosure is shown.

[0021] Figure 10 A schematic structural diagram of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0023] This disclosure proposes extracting material from various types of data related to an object, finding the textual descriptions corresponding to the material, generating text paragraphs based on the textual descriptions, and combining the text paragraphs to generate text describing the object. Thus, by displaying or broadcasting this text to the user, the user can better understand the object without having to perform extensive search operations.

[0024] Figure 1 A schematic flowchart of a text generation method according to an embodiment of the present disclosure is shown. Figure 1 The method shown can be implemented in software by a computer program or executed by a specially configured computing device. Figure 1 The method shown.

[0025] See also Figure 1 , in step S110 , one or more source data related to the object are obtained.

[0026] Objects may include, but are not limited to, commodities, applications, audiovisual programs (such as music, movies, TV series, novels), and other types of objects.

[0027] Source data can be any type of object-related data obtained through, but not limited to, web crawling, data sharing, or other data acquisition methods. Specifically, it can include data in multiple modalities (i.e., multiple types) such as text, images, audio, and video. For example, the acquired source data may include images, text, images (videos, photos, animations, etc.), sounds, and combinations thereof.

[0028] In step S120 , the source data is analyzed to obtain one or more pieces of material information.

[0029] Source data can be considered as raw data directly obtained without processing. Material information can be considered as data obtained by further analyzing the source data, that is, data extracted from the source data.

[0030] From the perspective of data sources, source data can include: unprocessed source data; and processed data that has undergone predetermined analysis and processing. Unprocessed source data refers to the data in the source data that can be directly used as source data. Processed data that has undergone predetermined analysis and processing refers to the new data obtained by analyzing the source data.

[0031] From the perspective of data function, material information refers to materials that are useful for generating texts used to describe objects. Specifically, it can include: materials that can provide content for text generation, that is, the materials corresponding to the text paragraph information on which the text generation described below depends; and materials that can be used to guide the output order of the content in the generated text, such as the materials that depend on constructing a text output framework that can guide the output order of the content in the text as described below.

[0032] From the perspective of data type, the material information may include but is not limited to at least one of text material, audio material, picture material and video material.

[0033] The process of analyzing source data to obtain material information can be considered as the process of mining the source data to enrich the material information. Depending on the type of source data, the method of analyzing the source data to obtain material information is also different.

[0034] Specifically, for source data of the image type, the first material information including the image and text information can be obtained by identifying the text information in the image. In particular, the text content in the image can be identified using OCR (Optical Character Recognition) technology.

[0035] For text-type source data (such as a first text), second material information including a second text for describing an object can be obtained based on the first text. The second text can be obtained based on the first text using a text generation technology.

[0036] For source data of audio and video types, the audio and video data can be analyzed to obtain third material information including at least one of the audio and video data, the voice recognition results of the audio and video data, and the audio and video analysis data. The audio and video data can be video data and / or audio data. The audio and video analysis data can include but is not limited to the number of plays, clicks, forwardings, interactions, comments, and viewing users of the audio and video data. Taking live broadcast data with audio and video data as the object as an example, the audio and video analysis data can also include the correlation between the live broadcast content at different times and the indicator data at the corresponding time (such as the number of product clicks, the number of orders, etc.). The quality of the indicator data can reflect the importance (or attractiveness, popularity) of the live broadcast content at the corresponding time. Specifically, the quality of the indicator data is positively correlated with the importance (or attractiveness, popularity) of the corresponding live broadcast content.

[0037] In step S130 , the material information is parsed to obtain corresponding one or more text paragraph information.

[0038] Text paragraph information can be considered as the textual description corresponding to the material information. Each text paragraph information can be composed of one or more sentences. As described above, the material information can include, but is not limited to, at least one of text material, audio material, image material, and video material. Different methods can be used to obtain the text paragraph information corresponding to different types of material information.

[0039] Specifically, for text materials, the text materials can be directly used as text paragraph information, or the text obtained after further processing of the text materials can be used as text paragraph information; and / or for audio materials, voice recognition can be performed on the audio materials to obtain text paragraph information corresponding to the audio materials; and / or for picture materials and / or video materials, semantic understanding can be performed on the picture materials and / or video materials to obtain text paragraph information corresponding to the picture materials and / or video materials.

[0040] For example, based on image caption technology and / or visual story generation technology, semantic understanding of picture materials can be performed to obtain text paragraph information used to describe the content of the picture materials; based on visual story generation technology, semantic understanding of multiple video frames in video materials can be performed to obtain text paragraph information used to describe the content of multiple video frames.

[0041] Image description generation technology integrates computer vision, natural language processing, and machine learning technologies to translate images into descriptive text. The implementation principles of image description generation technology can be found in the prior art and will not be elaborated in this disclosure.

[0042] Visual story generation technology is a technology that generates corresponding story content from images, image sequences, and video frame sequences. It can generate a coherent story from image sequences. The implementation principles of visual story generation technology can be found in the existing technology, and this disclosure will not elaborate on them.

[0043] In step S140 , text describing the object is generated using the text paragraph information.

[0044] The text paragraph information can be combined to generate the final text describing the object. For example, the text paragraph information can be reorganized to obtain the final text describing the object. Content reorganization refers to the arrangement and reorganization of the text paragraph information based on the given text paragraph information to enrich the text content diversity.

[0045] For example, a pre-trained text generation model can be used to generate text for introducing an object based on text paragraph information. The text generation model can be, but is not limited to, an Encoder-Decoder model, which may include an encoding part (encoder) and a decoding part (decoder). The encoding part is used to encode the input into a vectorized representation, and the decoding part is used to read the vectorized representation and output text. Among them, the input of the text generation model may include, but is not limited to, the name of the object, the attribute information of the object (such as the attribute name and attribute value of the object), and the text paragraph information. Thus, the name of the object, the attribute information of the object, and the text paragraph information can be input into a pre-trained text generation model to obtain text.

[0046] An object may include one or more attributes, and an attribute is used to represent the characteristics of the object. The characteristics represented by the attribute may include, but are not limited to, the appearance characteristics, functional characteristics, user characteristics, application scenario characteristics, etc. of the object. Depending on the type of object, the attributes of the object and the characteristics represented by the attributes are also different. Taking the object as a commodity as an example, the attributes of the object may include, but are not limited to, the category to which the commodity belongs (such as commodity brand, classification, basic style, applicable objects, usage scenarios, main material content, style details, etc.), sales volume, price, popularity, praise rate, etc.

[0047] As an example, before using the text paragraph information to generate the text describing the object (i.e., step S140), some of the text paragraph information can be selected from the obtained text paragraph information. Thus, when executing step S140, the selected text paragraph information can be used to generate the text describing the object. Specifically, multiple text paragraph information can be selected based on external input. For example, after obtaining the text paragraph information, the text paragraph information can be displayed to the user, and the text paragraph information participating in the text generation can be selected based on the selection operation made by the user.

[0048] The present disclosure can also establish a text output framework according to preset rules, and determine the output order of text paragraph information based on the text output framework to generate text. Thus, by using the text output framework to guide the output order of text paragraph information, and using the text paragraph information to generate text describing the object according to the output order of the text paragraph information, the content output order (i.e., content layout) of the generated text can be made more reasonable. Optionally, the established text output framework can also be adjusted to make the text output framework more accurate. The output framework can be adjusted based on external input, for example, the established text output framework can be displayed to the user, and the user can adjust the text output framework, such as the user can adjust the obviously unreasonable parts in the text output framework based on his own experience.

[0049] As an example, a text output framework can be established based on at least a portion of the source information. This established text output framework can be used to guide the output order of text paragraph information derived from the source information. The source information derived from analyzing the source data can be divided into two parts based on their function: one part is used to provide content for text generation, i.e., text paragraph information; the other part is used to construct the text output framework. These two parts of material may overlap.

[0050] The text output framework can be used to indicate the output order of the attributes of an object. When determining the output order of text paragraph information based on the text output framework, the attribute to which the text paragraph information is related can be determined by, but not limited to, the mode of semantic analysis of the text paragraph information, and then the output order of the attribute indicated by the text output framework is used as the output order of the text paragraph information. Wherein, when a single text paragraph information involves a plurality of attributes, the cumulative result of the output order of the plurality of attributes indicated by the text output framework can be used as the output order of the text paragraph information. Alternatively, priorities can also be set for different attributes. When a single text paragraph information involves a plurality of attributes, the cumulative result of the output order of the plurality of attributes indicated by the text output framework under corresponding weights can be used as the output order of the text paragraph information. Alternatively, the output order of the attribute with the highest priority among the plurality of attributes indicated by the text output framework can be used as the output order of the text paragraph information.

[0051] When analyzing source data to obtain one or more material information, one or more attributes of the source data can be analyzed simultaneously, and a text output framework can be established based on the corresponding attributes. For example, a text output framework for indicating the attribute output order can be established based on the attribute occurrence frequency and / or attribute retrieval popularity. The priority of the attribute output order is positively correlated with the attribute occurrence frequency and / or the attribute retrieval popularity. The attribute occurrence frequency refers to the frequency of occurrence of the attribute in the material (or source data), and the attribute retrieval popularity can be determined based on the historical retrieval records of one or more users.

[0052] Considering that object attributes are somewhat correlated in the output, highly correlated attributes often appear within the same text paragraph. Therefore, we can also calculate the similarity between materials with different attributes, merging the attributes corresponding to materials with similarities greater than a threshold into a single attribute. Similarity can refer to the degree of semantic or string matching between materials.

[0053] For example, the similarity between materials with different attributes can be calculated using, but not limited to, the Deep Structured Semantic Model (DSSM) and / or the Longest Common Subsequence (LCS). DSSM is used to calculate the degree of semantic matching, while LCS is used to calculate the degree of string matching. For more information about DSSM and LCS, please refer to the prior art.

[0054] Calculating the similarity between materials with different attributes can mean treating all materials related to the attribute as a whole and then calculating the similarity between materials with different attributes. Attributes corresponding to materials with similarity greater than a threshold are merged into one attribute. In other words, when determining the output order of attributes, the attributes corresponding to materials with similarity greater than the threshold are considered as one attribute.

[0055] For example, assuming that the material for attribute A includes sentence A1 and sentence A2, and the material for attribute B includes sentence B1 and sentence B2, if the similarity calculation shows that "sentence A1+A2" and "sentence B1+B2" are more similar, then attribute A and attribute B can be considered as similar attributes. When determining the output order of attribute A and attribute B, attribute A and attribute B can be merged into one attribute C, and the calculated output order of attribute C can be used as the output order of attribute A and attribute B.

[0056] As an example, the material information obtained by analyzing the source data may include audio material and / or video material. Considering that the text output order in the audio material and video material can reflect the output order of the text in the real scene to a certain extent, the present disclosure can use the audio material and / or video material to adjust the text output framework. For example, the text output framework can be adjusted based on the speech recognition results obtained by performing speech recognition on the audio material, and / or the video analysis data obtained by analyzing the video material. Among them, the video analysis data is also the audio and video analysis data mentioned above. Taking the audio and video data as live broadcast data as an example, the audio and video analysis data can not only be used to help construct a text output framework with an output order that conforms to the logic of the live broadcast scene, but also enable the output order of the constructed text output framework to reflect the importance (or attractiveness, popularity) of the content.

[0057] The present disclosure can also use pre-generated transition sentence templates to connect text paragraph information. Thus, by using transition sentence templates to guide the connection of text paragraph information, the transition logic between text paragraph information can be determined or adjusted to make the generated text context more fluent and natural.

[0058] Optionally, a pre-generated transition sentence template can be used to set (or adjust, modify) the transition logic (i.e., concatenation logic) between different text paragraph information (or different sentences) in the text. For example, the transition sentence template can be used to determine the transition sentences between different text paragraph information. The transition sentence is also the sentence used to connect (or concatenate) the context. The transition sentence template can help the concatenation logic between sentences to distinguish the content theme introduced by the context, while the manual writing method requires a lot of manpower. Therefore, the use of the transition sentence template can greatly reduce human resource consumption to a certain extent. Among them, the transition sentence template can be trained using natural language generation technology and machine learning technology. For example, the transition sentence template can be trained using sentence pairs as sample input and the concatenation logic between sentences (such as transition sentences) as labels. The specific training process will not be repeated in this disclosure.

[0059] As an example, the present disclosure can also select one or more pieces of material information and one or more corresponding text paragraphs, convert the text paragraphs into voice information, combine the material information and voice information into demonstration data, and present the demonstration data on the user interface. Thus, the present disclosure can be used to automatically generate an introduction video for an object, and play it with voice in conjunction with the text paragraph information.

[0060] Figure 2 A schematic flowchart of a text generation method according to another embodiment of the present disclosure is shown.

[0061] like Figure 2 As shown, first, in step S1, the source data related to the object can be analyzed to obtain material information based on the source data. The source data and the process of obtaining material information based on the source data can be found in the above description and will not be repeated here.

[0062] In step S2, after obtaining the material information, the material information can be parsed to obtain textual description information of the material information. This textual description information is also referred to as the text paragraph information mentioned above. The textual description information can be considered the result of textual parsing of the material information and is used to describe the material information in textual form. The parsed material information refers to the material used to provide content for text generation. The process of obtaining textual description information based on the material information can be found in the relevant description above and will not be repeated here.

[0063] In step S3, a text for describing the object is generated based on the text description information.

[0064] The text description information mentioned here may refer to the text description information obtained by parsing various types of material information such as text material, audio material, picture material, and video material. The generated text may include one or more paragraphs.

[0065] like Figure 3As shown by the dashed line, an output framework indicating the output order of the content can also be constructed based on at least part of the material information, and the output order of the content in the text can be determined by using the output framework. The output framework, i.e., the text output framework mentioned above, can refer to the description of the text output framework above, and will not be repeated here.

[0066] After the output order of the sentences in the text is determined based on the output framework, the text can be broadcasted according to the determined output order, or the content in the text can be laid out according to the determined output order, and the text can be displayed in a visual manner.

[0067] The text generation method of the present disclosure can also start from step S2. That is, the material information related to the object can be directly obtained, the text description information of the material information can be obtained by analyzing the material information, and then the text describing the object can be generated based on the text description information. Details related to the method can refer to the description above, and the present disclosure will not be repeated here.

[0068] Taking application in the e-commerce field as an example, the present disclosure can be implemented as a text generation method for a product to generate a text of the product, comprising: obtaining one or more source data related to the object; analyzing the source data to obtain one or more material information; analyzing the material information to obtain corresponding one or more text paragraph information; and generating a text describing the object by using the text paragraph information. Details related to the method can refer to the description above, and the present disclosure will not be repeated here.

[0069] Taking application in the e-commerce field as an example, the present disclosure also provides a live broadcast method, comprising: obtaining a text for introducing a product; and broadcasting the text in the process of playing a live broadcast picture related to the product. The broadcasted text can be a text obtained based on the text generation method of the present disclosure. The broadcasting of the text can be completed by a real anchor or a virtual anchor.

[0070] The present disclosure also provides an information display method, comprising: displaying a plurality of selection items on the screen of an electronic device, each selection item corresponding to a product; displaying a detail page of a product selected by a user in response to a selection operation of the user; and broadcasting a text for introducing the product. The broadcasted text can be a text obtained based on the text generation method of the present disclosure.

[0071] The text for introducing the product can be automatically broadcasted at the same time or after the detail page of the product is opened. Alternatively, as shown in FIG. 8, a play button can be set on the detail page of the product, and the text for introducing the product can be broadcasted in response to the user clicking the play button. Figure 3

[0072] The details related to the present disclosure will be further described below in conjunction with specific embodiments.​

[0073] Figure 4 A schematic diagram of the script construction principle according to an embodiment of the present disclosure applied to the field of e-commerce live streaming is shown.

[0074] like Figure 4 As shown, the present disclosure can obtain data of various modalities related to the object, such as main pictures / details pages, attribute / selling point knowledge, promotional articles, audio and video data, and generate a script suitable for broadcasting to describe the object by executing the above-mentioned text generation method.

[0075] The script can be text or an introductory video with voice broadcast accompanied by text. By showing or broadcasting the script to the user, the user can fully understand the object. The main picture / details page refers to the main carrier page of the object's descriptive information. For example, it can be the most important picture among multiple pictures related to the object, that is, the picture that contains more information. Selling points refer to the characteristics or attributes of an object that can attract consumers, such as the object's function, price, sales volume, etc. Selling point knowledge can refer to the knowledge set related to the object's selling points, and attribute knowledge can refer to the knowledge set related to the object's attributes. For information about attributes and audio and video data, please refer to the relevant description above and will not be repeated here.

[0076] Taking commodities as an example, we can process various modal data related to commodities, such as the commodity's main picture, commodity profile page, attribute / selling point knowledge, promotional articles, and live broadcast data, and convert them into text that conforms to the logic of the live broadcast scene, or a video for introducing the commodity.

[0077] Taking an application as an example, we can process various modal data related to the application, such as application charts, application screens, promotional videos, slogans, download volumes, and player comments, and convert them into text that can fully introduce the application. By displaying or broadcasting this text to users who have not installed the application, users can fully understand the application.

[0078] For other types of objects, the text used to introduce the object obtained by the text generation method of the present disclosure can be output to the user through visual display or voice broadcast, which can also enable the user to fully understand the object. The application scenarios of other types of objects will not be described in detail in this disclosure.

[0079] Considering that an object (such as a product) has multiple modal information such as text, pictures, and videos, in order to obtain text that can fully introduce the object, it is necessary to analyze the multimodal information of the object.

[0080] Analyzing multimodal information typically requires significant human and material resources, such as those required to research the subject and process and edit the various modal data related to the subject. Automatically processing the subject's multimodal data and converting it into text that comprehensively describes the subject (e.g., a script suitable for broadcast that conforms to the logic of a live broadcast scenario, or a film introducing the subject) would save significant manpower and allow for highly efficient utilization of multimodal materials, increasing the richness of information while also enriching the presentation.

[0081] However, constructing texts based on multimodal information is a challenging task. This is because the source data upon which texts are constructed is often long, has diverse sources, and is rich in multimodal information. Furthermore, relying on manually customized and manually screened solutions to process multimodal data often fails to fully utilize the interdependencies between these multimodal data.

[0082] The text generation method disclosed herein can automatically construct text for introducing an object based on the multimodal data of the object without requiring a large amount of manual intervention. Figure 5 A schematic flowchart of constructing a script based on multimodal data of an object according to an embodiment of the present disclosure is shown.

[0083] like Figure 5 As shown, the source data may include: main image / product details page; attribute / selling point knowledge, promotional articles; and live broadcast data.

[0084] For main images / product detail pages, OCR technology can be used to identify text within images, yielding "image + text" graphic materials. For attribute / selling point knowledge and promotional articles, descriptive statements describing the attributes and selling points of an object can be obtained. For live streaming data, analysis can yield video frame data, ASR data, and live streaming replay analysis data. Video frame data refers to the video frames included in the live video data within the live streaming data, ASR data refers to the speech recognition results obtained by performing speech recognition on the live audio data within the live streaming data using ASR (Automatic Speech Recognition) technology, and live streaming replay analysis data refers to the analysis results obtained by replaying and analyzing the live streaming data. Live streaming replay analysis data is primarily used to construct an output framework, so it can be data that can guide the order of content output. For example, live streaming replay analysis data can include analytical data on the correlation between one or more indicators, such as live streaming views, comments, reposts, followers, product clicks, orders, and payments, and the live streaming content. Optionally, the live broadcast replay analysis data may include the correlation between the live broadcast content at different times and the indicator data at the corresponding times (such as the number of object clicks, the number of orders, etc.). The quality of the indicator data can reflect the importance (or attractiveness, popularity) of the live broadcast content at the corresponding time. Specifically, the quality of the indicator data is positively correlated with the importance (or attractiveness, popularity) of the corresponding live broadcast content.

[0085] Therefore, the materials obtained based on the source data by analyzing the source data may include: graphic information, sentences used to describe the attributes / selling points of the object, video frames, ASR data, live broadcast replay analysis data and attribute knowledge.

[0086] Some of the aforementioned materials (such as image and text information, attribute / selling point sentences, and video frames) are used to provide content for text generation; other materials (such as attribute / selling point sentences, ASR data, live broadcast replay analysis data, and attribute knowledge) are used to build the broadcast framework. The broadcast framework mentioned here refers to a framework that conforms to the broadcast logic of the live broadcast scenario.

[0087] 1. Text generation process

[0088] The materials used to provide content for text generation can include text materials, image materials, and video materials. For text materials, they can be left unprocessed for the time being. That is, text materials can directly participate in content reorganization to generate text.

[0089] For picture materials, image description generation technology can be used to obtain text description information for the picture (corresponding to the first text description information mentioned above), and the text description information can be a sentence description.

[0090] For image and video materials, visual story generation technology can also be used to generate and obtain text description information in the image / video frame through semantic understanding of the image / video frame to describe and introduce the main object targeted by the image / video frame.

[0091] For example, for Figure 6 The following text can be obtained by using visual story generation technology from the series of pictures shown: "The collar of this dress is a hand-knitted round collar with fine workmanship. The bottom edge of the body is relatively flat. The buttons are sewn by hand and fit the body well without feeling abrupt. The cuffs are loose and you won't feel stuffy even in hot weather. The pattern on the body is very beautiful."

[0092] Then, the text material and the text description information obtained based on the picture material and video material can be reorganized to enrich the diversity of the text content.

[0093] Figure 7 A schematic diagram of the content reorganization principle according to an embodiment of the present disclosure is shown.

[0094] like Figure 7 As shown, the object name, object attributes, and multiple description texts can be input into a text generation model including an encoder and a decoder to obtain segmented text.

[0095] The object attribute may refer to an attribute involved in multiple description texts. If multiple description texts do not involve the object attribute, the input object attribute may be a null value.

[0096] Multiple description texts may include: text information obtained by OCR detection of the main image / details page, descriptive sentences obtained based on attribute / selling point knowledge, descriptive sentences obtained based on pictures / videos, etc.

[0097] Example 1:

[0098] Multiple description texts can include sentences like:

[0099] Sentence 1: The round neck design has the effect of lengthening the neck;

[0100] Sentence 2: The medium-length version can be paired with a skirt;

[0101] Sentence 3: The short-sleeved design is refreshing and pleasant, while making the arms look slender and beautiful.

[0102] The output obtained by the encoder and decoder is the following text:

[0103] You can see that this dress has a round neck design, which will have a lengthening effect on the neck; you can also see that it has short sleeves, so you won’t feel stuffy when wearing it; the medium-long length of the dress can also be used as a skirt.

[0104] Example 2:

[0105] Multiple description texts can include sentences like:

[0106] Sentence 1, the design of bright silk elements highlights the individual femininity;

[0107] Sentence 2: The embellishment of bright silk exudes a different style;

[0108] Sentence 3: The holes are carefully cut to show off the charming posture;

[0109] Sentence 4: Personalized and creative hole design, very fashionable and trendy;

[0110] Sentence 5: The hollow hole design on the clothes can also easily bring a more fashionable look.

[0111] The output obtained by the encoder and decoder is the following text:

[0112] This sweater is embellished with shiny silk, exuding a unique style and highlighting individual femininity. The hollow and ripped design keeps up with fashion trends, making you look charming when you wear it.

[0113] 2. Broadcast framework construction process

[0114] A broadcast framework (corresponding to the output framework described above) can be constructed based on the ASR data, live broadcast replay analysis data, and materials used to describe product attributes in the material information. Materials used to describe product attributes can include descriptive statements (such as attribute / selling point sentences) derived from attribute / selling point knowledge and attribute knowledge. Attribute knowledge can refer to a collection of knowledge related to object attributes.

[0115] Figure 8 A schematic diagram of the construction process of a broadcast framework according to an embodiment of the present disclosure is shown.

[0116] like Figure 8 As shown, the attribute reporting order (corresponding to the output order mentioned above) can be determined based on the attribute occurrence frequency in the material (attribute / selling point sentences and attribute knowledge) and / or the search popularity corresponding to the attribute. For example, the attribute reporting order can be obtained by comprehensively scoring the attribute occurrence frequency in the material and the search popularity of the attribute in the material. Among them, the priority of the attribute reporting order is positively correlated with the attribute occurrence frequency and the attribute search popularity; the search popularity corresponding to the attribute can be determined based on the historical search records of multiple users.

[0117] Considering that the attributes of the object have certain relevance in the output, the attributes with strong relevance are mostly described in the same sentence or paragraph. The disclosure can also calculate the similarity between the materials of different attributes, and merge the attributes corresponding to the materials with a similarity greater than a threshold into one attribute. The similarity can refer to the degree of semantic matching or string matching between the materials.

[0118] According to the speech recognition result of the broadcast audio data, the broadcast order of the broadcast content can be obtained, which can be regarded as a content broadcast order conforming to the broadcast scene logic. The broadcast playback analysis data obtained by analyzing the live broadcast data can reflect the importance of different broadcast contents.

[0119] Therefore, the speech recognition result of the broadcast audio data and the broadcast playback analysis data of the broadcast data can not only be used to help construct a broadcast order conforming to the broadcast scene logic, but also make the constructed broadcast order reflect the importance (or attractiveness, popularity) of the content. Thus, the broadcast order of the attributes can be adjusted by analyzing the speech recognition result and the broadcast playback analysis data.

[0120] After obtaining the broadcast framework, the broadcast order of the content in the text can be determined using the broadcast framework. As shown in FIG. 8, the text can also be guided by a stringing technique using a pre-generated transition sentence template to determine (or adjust) the transition logic between sentences in the text. Figure 5

[0121] Thus, the final script suitable for broadcasting conforming to the broadcast scene logic can be obtained.

[0122] The disclosure supports multiple modalities of data input and supports automatic processing of multiple modalities of data. Specifically, the disclosure designs a complete process to utilize multi-source multi-modal data, and according to the method of modal conversion, maps various modalities to a certain modality script, and realizes multi-modal display through matching and mounting, and constructs a complete process from multi-modal input to multi-modal display; combines multi-modal data statistics and algorithms and applies them to the broadcast content construction process to guide the construction of content conforming to the live broadcast scene characteristics.

[0123] The text generation method of the disclosure can also be implemented as a text generation device. Figure 9 The structure schematic diagram of the text generation device according to the exemplary embodiments of the disclosure is shown. The functional units of the text generation device can be realized by hardware, software or a combination of hardware and software that implements the principles of the disclosure. Those skilled in the art can understand that the functional units of the text generation device can be implemented by one or more of the following means: Figure 9 ​The functional units described can be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein can support any possible combination, division, or further limitation of the functional units described herein.

[0124] The following is a brief description of the functional units that a text generation device may have and the operations that each functional unit may perform. For the details involved, please refer to the relevant description above and will not be repeated here.

[0125] See also Figure 9 The text generation device 900 includes an acquisition module 910 , an analysis module 920 , a parsing module 930 and a generation module 940 .

[0126] The acquisition module 910 is used to acquire one or more source data related to the object. The analysis module 920 is used to analyze the source data to obtain one or more material information. The parsing module 930 is used to parse the material information to obtain one or more corresponding text paragraphs. The generation module 940 is used to generate text describing the object using the text paragraph information.

[0127] The generation module 940 may include a creation module and a determination module. The creation module is used to create a text output framework according to a preset rule. The determination module is used to determine the output order of text paragraph information based on the text output framework to generate text.

[0128] The text generating apparatus 900 may further include an adjustment module for adjusting the output framework based on external input.

[0129] The analysis module 920 can analyze one or more attributes of the source data. The text generation device 900 can also include a calculation module and a merging module. The calculation module is used to calculate the similarity between materials with different attributes, and the merging module is used to merge the attributes corresponding to materials with similarities greater than a threshold into a single attribute.

[0130] The establishing module can establish a text output framework according to the corresponding attributes. The determining module can output the text paragraph information according to the output sequence.

[0131] The establishment module can establish a text output framework for indicating the attribute output order according to the attribute occurrence frequency and / or attribute search popularity. The adjustment module can also adjust the text output framework using audio materials and / or video materials.

[0132] The text generating device 900 may further include a connection module for connecting the text paragraph information using a pre-generated linking sentence template.

[0133] The text generation device 900 may further include a selection module for selecting part of the text paragraph information. The generation module 940 may use the selected text paragraph information to generate a text describing the object.

[0134] The generation module 940 may input the name of the object, the attribute information of the object, and the text paragraph information into a pre-trained text generation model to obtain the text.

[0135] The analysis module 920 can obtain first material information including the picture and the text information by identifying the text information in the picture; and / or obtain second material information including a second text used to describe the object based on the first text; and / or obtain third material information including video data, speech recognition results of audio and video data, and at least one of audio and video analysis data by analyzing the audio and video data.

[0136] The parsing module 930 can obtain text paragraph information corresponding to the audio material by performing speech recognition on the audio material; and / or obtain text paragraph information corresponding to the picture material and / or the video material by performing semantic understanding on the picture material and / or the video material.

[0137] As an example, the generation module 940 can select one or more material information and corresponding one or more text paragraph information, convert the text paragraph information into voice information, and combine the material information and voice information into demonstration data. The text generation device 900 can also include a presentation module for presenting the demonstration data on a user interface.

[0138] Figure 10 A schematic diagram of the structure of a computing device that can be used to implement any one of the above-mentioned text generation method, live broadcast method, and information display method according to an embodiment of the present invention is shown.

[0139] See also Figure 10 , the computing device 1000 includes a memory 1010 and a processor 1020 .

[0140] Processor 1020 may be a multi-core processor or may include multiple processors. In some embodiments, processor 1020 may include a general-purpose main processor and one or more specialized coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, processor 1020 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0141] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0142] The memory 1010 stores executable code. When the executable code is processed by the processor 1020, the processor 1020 can execute any one of the text generation method, broadcasting method, and information display method mentioned above.

[0143] The text generation method, live broadcast method, information display method, related devices and computing equipment according to the present invention have been described in detail above with reference to the accompanying drawings.

[0144] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0145] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0146] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0147] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A text generation method for generating text related to an object, comprising: Acquire a plurality of source data related to an object, wherein the plurality of source data includes multimodal data of the object; Analyzing the source data to obtain one or more pieces of material information; Parsing the material information to obtain corresponding one or more text paragraph information; Generating a text describing the object using the text paragraph information, wherein an output order of the text paragraph information in the text is determined based on a text output framework, the text output framework being used to indicate an output order of attributes of the object; Among them, the text output framework is determined in the following manner: determining at least one attribute corresponding to the multiple source data, and establishing an initial text output framework based on the at least one attribute; determining audio material information and / or video material information from the material information; determining the text output order in the audio material information and / or video material information; adjusting the initial text output framework based on the text output order to obtain the text output framework.

2. The method according to claim 1, further comprising: Select one or more material information and corresponding one or more text paragraph information; Convert text paragraph information into voice information; Combining material information and voice information into presentation data; Present demonstration data in the user interface.

3. The method according to claim 1, wherein Generating text describing the object using the text paragraph information includes: Establish a text output framework based on preset rules; Based on the text output framework, an output order of the text paragraph information is determined to generate text.

4. The method according to claim 3, further comprising: The output framework is adjusted based on external input.

5. The method according to claim 3, wherein Analyzing the source data to obtain one or more material information includes: Analyze one or more attributes of the source data.

6. The method according to claim 5, further comprising: Calculate the similarity between materials with different attributes; Merge the attributes corresponding to materials with similarity greater than a threshold into one attribute.

7. The method according to claim 5, further comprising: Establish a text output framework based on the corresponding attributes; and / or The text paragraph information is output according to the output order.

8. The method according to claim 7, wherein: Establishing a text output framework based on the corresponding attributes includes: A text output framework is established to indicate the order of attribute output according to attribute occurrence frequency and / or attribute retrieval popularity.

9. The method according to claim 7, further comprising: The text output framework is adapted using audio material and / or video material.

10. The method according to claim 1, further comprising: The text paragraph information is connected using a pre-generated linking sentence template.

11. The method according to claim 1 , further comprising: Selecting a portion of the text paragraph information, wherein using the text paragraph information to generate text describing the object includes: using the selected text paragraph information to generate text describing the object.

12. The method according to claim 1, wherein The step of generating text describing the object using the text paragraph information includes: The name of the object, the attribute information of the object, and the text paragraph information are input into a pre-trained text generation model to obtain the text.

13. The method according to claim 1, wherein The source data includes data of multiple modalities.

14. The method according to claim 13, wherein The source data includes pictures, text, images, sounds and combinations thereof.

15. The method according to claim 1, wherein The step of analyzing the source data to obtain one or more material information includes: Obtaining first material information including the image and the text information by identifying text information in the image; and / or Based on the first text, obtaining second material information including a second text for describing the object; and / or By analyzing the audio and video data, third material information including at least one of the video data, the speech recognition result of the audio and video data, and the audio and video analysis data is obtained.

16. The method according to claim 1, wherein The material information includes at least one of text material, audio material, picture material and video material.

17. The method according to claim 16, wherein Parsing the material information to obtain corresponding one or more text paragraph information includes: Obtaining text paragraph information corresponding to the audio material by performing speech recognition on the audio material; and / or By performing semantic understanding on the picture material and / or the video material, text paragraph information corresponding to the picture material and / or the video material is obtained.

18. The method according to claim 1, wherein The material information includes: the unprocessed source data; and Processed data after preset analysis.

19. A text generation device for generating text related to an object, comprising: an acquisition module, configured to acquire a plurality of source data related to an object, wherein the plurality of source data includes multimodal data of the object; An analysis module, configured to analyze the source data to obtain one or more pieces of material information; a parsing module, configured to parse the material information to obtain corresponding one or more text paragraph information; and A generation module is used to generate text describing the object using the text paragraph information, wherein the output order of the text paragraph information in the text is determined based on a text output framework, and the text output framework is used to indicate the output order of the attributes of the object; the text output framework is determined in the following manner: determining at least one attribute corresponding to the multiple source data, and establishing an initial text output framework based on the at least one attribute; determining audio material information and / or video material information from the material information; determining the text output order in the audio material information and / or video material information; and adjusting the initial text output framework based on the text output order to obtain the text output framework.

20. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 18.

21. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method according to any one of claims 1 to 18.

Images