Article and newsprint generation method and device, storage medium and program product
By extracting multimodal data from target video clips and using a multimodal model to generate a draft article, the problem of AI writing failing to focus on the topic has been solved, achieving higher accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2026-03-10
AI Technical Summary
Existing AI writing technology cannot effectively focus on the topics expected by users when generating articles, resulting in low accuracy. Furthermore, the original videos contain multiple topics, leading to errors or omissions in information expression.
By extracting multimodal data from target video clips corresponding to the topic to be written, a first draft of the article is generated using a multimodal model, and then combined with the writing requirements of natural language description to generate the first draft of the article.
It improves the accuracy of article generation, focuses on the topics users expect, provides more reference information, and enhances the accuracy and efficiency of AI-generated articles.
Smart Images
Figure CN121640331A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium, and program product for generating articles and press releases. Background Technology
[0002] With the continuous development of internet technology, various media channels have emerged for people to publish articles, such as news dissemination via the internet, improving the efficiency and reach of information dissemination. A large number of videos are generated online every day, and some key points in these videos deserve widespread dissemination or reporting. One method of dissemination is to compose an article based on the video content and then spread or report on it.
[0003] To improve article production efficiency, some self-media creators are using artificial intelligence (AI) technology to generate articles based on video understanding. However, the accuracy of current AI-generated articles is relatively low. Summary of the Invention
[0004] This application provides a method, apparatus, storage medium, and program product for generating articles and press releases, thereby improving the accuracy of the generated articles.
[0005] This application provides an article generation method, including:
[0006] Extract the target video clips corresponding to the topic to be written from the original videos that provide writing material;
[0007] Obtain article writing requirements described in natural language;
[0008] Extract multimodal data from the target video segment;
[0009] A first draft of the article is generated using a multimodal model based on the multimodal data and the article writing requirements.
[0010] This application also provides a method for generating press releases, including:
[0011] Extract the target video clips corresponding to the topic to be written from the original videos that provide news materials;
[0012] Obtain news writing requirements described in natural language;
[0013] Extract multimodal data from the target video segment;
[0014] A first draft of the news is generated based on the multimodal data and the news writing requirements using a multimodal model.
[0015] This application also provides an electronic device, including: a memory and a processor; wherein the memory is used to store computer programs;
[0016] The processor is coupled to the memory for executing the computer program to perform steps in the article generation method and / or news generation method.
[0017] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform steps in an article generation method and / or a news generation method.
[0018] This application also provides a computer program product, including a computer program that, when executed by one or more processors, causes the one or more processors to perform steps in an article generation method and / or a news generation method.
[0019] In this embodiment, the multimodal model uses multimodal data from the target video segment corresponding to the topic to be written, rather than the entire original video, to generate a draft article. This allows the multimodal model to focus on the topic the user expects to write about (i.e., the topic to be written), improving the alignment between the generated article and the user's expectations, thus increasing the accuracy of the generated article. Furthermore, the multimodal data generated from the target video segment provides more reference information, enriching the dimensions of reference information for AI writing and making the generated article more accurate, thereby improving the accuracy of AI writing. Attached Figure Description
[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0021] Figure 1 A flowchart illustrating the article generation method provided in this application embodiment;
[0022] Figure 2 This application provides a schematic diagram illustrating the human-computer interaction process during article generation.
[0023] Figure 3 A schematic diagram illustrating the process of determining the target time period provided in an embodiment of this application;
[0024] Figure 4 A schematic diagram illustrating the process of obtaining article writing requirements as provided in this application embodiment;
[0025] Figure 5The illustrations show the display effect of the article in the human-computer interaction mode and the final draft confirmation effect, as provided in the embodiments of this application.
[0026] Figure 6a and Figure 6b A schematic diagram illustrating the final draft of the article provided in this application embodiment;
[0027] Figure 7 A flowchart illustrating the press release generation method provided in this application embodiment;
[0028] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0031] A vast amount of video is generated online every day, and some of these videos contain key points that deserve widespread dissemination or reporting. For example, videos or recordings of international sporting events may contain memorable moments or outstanding individuals that deserve broad coverage. One approach to reporting on these key points is to write articles highlighting them.
[0032] In some traditional AI-powered article writing solutions, self-media creators first download the original video to their local terminal device, then use speech recognition software to convert the audio in the original video into a text transcript. After that, they optimize the text transcript to remove any errors, and then put the optimized text into the AI writing software, which then outputs the corresponding article.
[0033] First, because AI-generated text writing software is fed only the text content of videos and cannot incorporate other information from the video, the output article may contain errors or omit key information, resulting in low accuracy. Second, the original video may cover many topics, and the creator may only want to write an article on one of them. Inputting the entire video's text into the AI writing software makes it difficult to focus on the intended topic, again leading to low accuracy.
[0034] To improve the accuracy of article generation, in some embodiments of this application, the multimodal model utilizes multimodal data from target video segments corresponding to the topic to be written, rather than the entire original video, to generate a draft article. This allows the multimodal model to focus on the topic the user expects to write about (i.e., the topic to be written), helping to improve the alignment between the generated article and the user's expectations, thus increasing the accuracy of the generated article. Furthermore, the multimodal model generates the draft article based on the multimodal data of the target video segments. This multimodal data provides more reference information for the generated draft article, enriching the dimensions of reference information for AI writing, making the article generated by the multimodal model more accurate, thereby improving the accuracy of AI writing.
[0035] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0036] It should be noted that the same reference numerals denote the same object in the following figures and embodiments. Therefore, once an object is defined in one figure or embodiment, it does not need to be discussed further in subsequent figures and embodiments.
[0037] Figure 1 This is a flowchart illustrating the article generation method provided in an embodiment of this application. Figure 1 As shown, the article generation method mainly includes:
[0038] 101. Extract the target video clips corresponding to the topic to be written from the original videos that provide writing materials.
[0039] 102. Obtain the article writing requirements described in natural language.
[0040] 103. Extract multimodal data from the target video segment.
[0041] 104. Generate the first draft of the article based on multimodal data and article writing requirements using a multimodal model.
[0042] The article generation method provided in this application can be deployed on any computing device. For example, the article generation method provided in this application can also be deployed on a terminal device or a server device. Preferably, the article generation method provided in this application can also be deployed on a server device, which can save the computing and storage resources of the user terminal. In particular, the multimodal model provided in this application requires a large amount of computing resources to run, and the user's terminal device may not be able to meet the computing resource requirements of the multimodal model. Therefore, deploying the article generation method provided in this application on a server device is a preferred approach.
[0043] The server-side device can be a single server device, a cloud-based server array, or a virtual machine (VM) running in a cloud-based server array.
[0044] In the embodiment of the article generation method provided in this application, which is deployed on a server-side device, the original video providing writing material can be stored on the server or downloaded in real time from other servers. Users of this article generation method can watch the original video online or download it to their terminal device for viewing. Specifically, when watching the original video online, it can be played on the video player on the user's terminal device, eliminating the need to download it locally, thus saving storage resources and reducing download time.
[0045] Users can preview the original video to determine the topic to be written. For the apparatus or device that performs the article generation method, in step 101, the target video segment corresponding to the topic to be written can be extracted from the original video that provides writing material.
[0046] In some embodiments, combined with Figure 2 and Figure 3 The device executing the article generation method may provide a user interface (UI), which may include a time selection control for the original video. The user can determine the time segment corresponding to the topic to be written from the original video. The time segment refers to a relative time period within the original video, such as the 60th to 65th minute of the original video. Furthermore, the user can use this time selection control to determine the target time segment corresponding to the topic to be written.
[0047] In some embodiments, an automatic script writing control can be added to the video player of the user's terminal device, such as... Figure 2 The "AI-generated writing" shown. Combined with... Figure 2 and Figure 3When a user triggers the automatic writing control, the terminal device can automatically display the aforementioned user interface. Specifically, in response to the triggering operation of the automatic writing control, the terminal device can send a writing request to the server (in this embodiment, the device executing the article generation method); the device executing the article generation method can, in response to the writing request, provide a user interface to the terminal device, which includes a time selection control. The terminal device can display this user interface on a video player. For example, layers of the user interface can be displayed on the original video being played by the video player.
[0048] In this application embodiment, the specific implementation of the time selection control is not limited. In some embodiments, such as Figure 3 As shown, the time selection control can be implemented as a time selection button on the progress bar of the original video. Users can determine the start and end times of the target time period by triggering (clicking or dragging) the time corresponding to the progress bar. Figure 3 In this embodiment, the target time period selected by the user is the original video's time period of 60:17-63:22. In other embodiments, the time selection control can be implemented as an input box (not shown in the accompanying drawings), where the user can enter the start and end times of the target time period.
[0049] For devices that perform article generation methods, combined with Figure 2 and Figure 3 It can retrieve the time period selected based on the time selection control as the target time period corresponding to the topic to be written, that is, retrieve the target time period selected according to the topic to be written. Furthermore, it can extract video segments of the target time period from the original video as the target video segments.
[0050] Specifically, such as Figure 2 As shown, the device executing the article generation method can install and run an automatic writing assistant, which can be a computer program software entity that can extract video segments of a target time period from the original video as target video segments.
[0051] In other embodiments, the user provides a topic to be written in natural language. For example, a device performing an article generation method may provide a user interface that may include a topic control (not shown in the figures), through which the user can input the topic to be written in natural language. The topic to be written described in natural language may be a topic described in text or speech in natural language. As another example, a device performing an article generation method may output query information regarding the topic to be written, such as "Please confirm the topic to be written." or "Which topic do you want to write an article about?". Specifically, in response to the aforementioned writing request, the device performing the article generation method enters a human-computer interaction mode and outputs query information regarding the topic to be written.
[0052] Specifically, a query message regarding the topic to be written can be output in text form, such as displaying "Please confirm the topic to be written." or "Which topic would you like to write an article about?" The user can respond to this query message by sending the topic to be written in text or voice form. Accordingly, the device executing the article generation method can acquire the voice or text data based on the response to the query message regarding the topic to be written, as a description of the topic to be written in natural language.
[0053] For example, a device performing an article generation method might output a question about the topic to be written in voice form, such as "Please confirm the topic to be written." or "Which topic would you like to write about?" The user can respond to this question by sending the topic to be written in voice form. Accordingly, the device performing the article generation method can acquire the voice data based on the response to the question about the topic to be written, as a description of the topic to be written in natural language. Alternatively, the topic to be written can be identified from the voice data based on the response to the question about the topic to be written.
[0054] Based on the user-provided writing topic, semantic analysis can be performed on the video frames of the original video to determine the associated topic content. Optionally, multimodal data can be extracted from the original video. The modality of the data can be referred to as the data type, and may include text, images, audio, and video, etc. These different types of data are considered different modalities. In this embodiment, the multimodal data includes at least two modalities of image data, text data, audio data, and background sound data. Preferably, the multimodal data includes image data, text data, and audio data.
[0055] In this application, the specific implementation method for extracting multimodal data from video frames of the original video is not limited. In some embodiments, a text recognition model can be used to obtain subtitle data from the video frames of the original video, and a speech recognition model can be used to obtain audio data from the video frames of the original video; the subtitle data and audio data are separated from the original video to obtain the image data contained in the original video. Furthermore, the subtitle data, audio data, and image data extracted from the original video can be used as multimodal data.
[0056] The text recognition model can be an Optical Character Recognition (OCR) model or a neural network model, etc. The neural network model can be a Convolutional Neural Network (CNN) model, a Recurrent Neural Network (RNN) model, a Deep Neural Network (DNN) model, or a Feedforward Neural Network (FNN) model, etc.
[0057] Alternatively, multimodal models can be used to extract multimodal data from the video frames of the original video. Multimodal models can extract at least two modalities of data from the original video, including image data, audio data, caption data, and background audio data, as multimodal data.
[0058] The multimodal model can be a neural network model. In some embodiments, when the number of parameters in the multimodal model is large, for example, when the number of parameters in the multimodal model is in the millions, hundreds of millions, billions, or even more, the multimodal model can also be called a large multimodal model. In this application embodiment, a large language model is defined as a neural network model whose number of model parameters conforms to a preset parameter number range. The preset parameter number range corresponds to a very large number of model parameters, which can be in the millions, hundreds of millions, billions, or even more, and the specific value can be determined by AI domain standards.
[0059] In this application, the specific implementation architecture of the multimodal model is not limited. The multimodal model can adopt a licensed and open-source multimodal model architecture, or it can adopt a self-developed multimodal model architecture. In some embodiments, the multimodal model can use an open-source multimodal architecture. Such models do not require further training by the user. Such multimodal models can output information adapted to the input prompt word upon triggering the input prompt.
[0060] In some embodiments, the multimodal model can be a model that needs to be trained, such as a large language model that needs to be trained in two phases: pre-training and fine-tuning. In the pre-training phase, the model is trained on a large-scale, general video dataset, learning the basic structure of language and various common-sense concepts. Then, in the fine-tuning phase, the model is further trained on a smaller, more domain-specific dataset. Fine-tuning allows the model to better understand and generate language specific to that domain, thereby better performing specific tasks.
[0061] During the training phase of the multimodal model, video samples and pre-collected article samples written based on the video samples can be obtained. Then, with the goal of minimizing the loss function, the initial multimodal model is trained using the video samples to obtain the multimodal model. The loss function can be represented by the difference between the articles predicted by the multimodal model and the corresponding article samples during model training.
[0062] Among them, multimodal models can process data from multiple modalities and learn their common semantics from data from different modalities, thereby achieving understanding and generation between different modalities.
[0063] After obtaining the multimodal data, a multimodal model can be used to perform semantic analysis on the multimodal data to determine the thematic content associated with the video frames of the original video. Furthermore, based on the thematic content associated with the video frames of the original video, video frames corresponding to the topic to be written can be extracted from the original video as target video segments.
[0064] For AI-generated articles, users can also provide writing requirements. Combined with... Figure 2 and Figure 4 The user describes the writing requirements of the article in natural language. In some embodiments, the device performing the article generation method may provide a user interface that may include writing requirement provision controls, through which the user can input the article writing requirements in natural language. The article writing requirements described in natural language may be article writing requirements described in text or speech in natural language. For example, such as Figure 4 As shown, for the device executing the article generation method (specifically, the automatic writing assistant running on that device), in human-computer interaction mode, it can output prompts regarding article writing requirements in voice or text form, such as playing "Please enter your writing requirements." or "What are your requirements for the article you are writing?" Figure 4The message "I am your AI writing assistant. I can write articles for you. Tell me your requirements." allows users to respond to this inquiry by sending their writing requirements in voice or text form. Accordingly, the device executing the article generation method can acquire the voice or text data based on the inquiry regarding the writing requirements, as a description of the writing requirements in natural language. Alternatively, the writing requirements can be identified from the voice or text data based on the inquiry. For example, a speech recognition model can be used to perform semantic analysis on the voice data based on the inquiry regarding the writing requirements to determine the writing requirements. For text data, a text recognition model can be used to perform semantic analysis on the text data based on the inquiry regarding the writing requirements to determine the writing requirements. In summary, as described in step 102, the device executing the article generation method can acquire the writing requirements described in natural language. The method of users describing their writing requirements in voice form, compared to describing them in text form, improves user expression efficiency, thereby contributing to improved efficiency in subsequent article generation. In this embodiment of the application, users can describe the article writing requirements through text or voice, which makes the human-computer interaction highly flexible, especially voice interaction, which is more convenient than text interaction.
[0065] In this embodiment of the application, the specific content of the article writing requirements is not limited. Optionally, the article writing requirements may include: the genre, word count, language, language style, and / or the data modalities contained in the article. The genre of the article refers to its style or category, which may include: news, poetry, novel, essay, or drama, etc. The language style may be professional, humorous, or colloquial, etc. For example, such as... Figure 4 As shown, the article writing requirements are as follows: Please write a press release, no more than 500 words, using professional journalistic terminology, and including relevant images. The article genre is a press release, the word count is limited to 500 words, the language style is professional journalistic, and the data modality is text and images.
[0066] The device executing the article generation method can be pre-configured with default article writing requirements. These default requirements can include various types, such as article genre, word count, language, language style, and data modalities. If the user-provided article writing requirements do not include one or more of these types, the default requirements for those types will be used. For example, if the user-provided article writing requirements do not include language and language style requirements, the default language and language style will be used.
[0067] Since videos contain multimodal data, which provides users with information from different dimensions or from different dimensions to write articles, in this embodiment, to improve the accuracy of article generation, multimodal data can be extracted from the target video segment in step 103. For specific implementation methods of extracting multimodal data from the target video segment, please refer to the relevant content on extracting multimodal data from the original video, which will not be repeated here.
[0068] Furthermore, since the multimodal model can learn the common semantics of data from different modalities, thereby achieving understanding and generation between different modalities, in step 104, the multimodal model can generate a first draft of the article based on the multimodal data extracted from the target video clip and the article writing requirements described in natural language.
[0069] Specifically, such as Figure 2 As shown, the automatic writing assistant can input multimodal data and article writing requirements into a multimodal model. Within the multimodal model, a first draft of the article can be generated based on the multimodal data extracted from the target video clip and the article writing requirements described in natural language. Specifically, the multimodal model can extract features from the multimodal data to obtain feature vectors. It can also vectorize the article writing requirements to obtain feature vectors. Specifically, the multimodal model can segment the article writing requirements to obtain the multiple words contained within them. Then, the word vector layer in the multimodal model can convert these multiple words into word vectors. Finally, these word vectors can be aggregated to obtain the feature vector of the article writing requirements.
[0070] Subsequently, the feature vectors of the multimodal data and the feature vectors required for article writing can be fused using a multimodal model to obtain the target feature vector.
[0071] Specifically, a multimodal model can be used to concatenate the feature vectors of the multimodal data with the feature vectors required for article writing, thereby achieving feature fusion and obtaining the target feature vector. Alternatively, a multimodal model can use a multimodal attention mechanism or a multimodal fusion layer to fuse the feature vectors of the multimodal data with the feature vectors required for article writing, thus obtaining the target feature vector.
[0072] Furthermore, the multimodal model can generate a draft article based on the target feature vector. Specifically, in the multimodal model, the text content of the draft article is determined based on the target feature vector. The text content of the draft article may include: the article title and the main body. The article title may include a main title and / or multiple levels of headings. Multiple levels refer to two or more levels, such as: main title, first-level heading, second-level heading, third-level heading, or even more levels of headings. The main body of the article includes one or more paragraphs, with multiple meaning two or more.
[0073] Furthermore, in a multimodal model, images corresponding to the text content can be determined from the target video clip. Specifically, the multimodal model can determine images corresponding to each paragraph of the main text from the target video clip; and / or, the multimodal model can determine images corresponding to the title content from the target video clip, which can then be used as images for the entire article, and so on.
[0074] After that, the text and images can be formatted to obtain the first draft of the article.
[0075] In some embodiments, text content and accompanying images can be formatted according to a pre-set layout format in a human-computer interaction mode to obtain a draft of the article. Here, the layout format refers to the distribution characteristics of the article's page elements, reflecting the overall layout and structure of the article. The page elements of an article are the overall representation of its layout, reflecting its structure, and may include elements such as body text, titles, images, columns, footnotes, and chart descriptions.
[0076] In this embodiment, the multimodal model uses multimodal data from the target video segment corresponding to the topic to be written, rather than the entire original video, to generate a draft article. This allows the multimodal model to focus on the topic the user expects to write about (i.e., the topic to be written), improving the alignment between the generated article and the user's expectations, thus increasing the accuracy of the generated article. Furthermore, the multimodal data generated from the target video segment provides more reference information, enriching the dimensions of reference information for AI writing and contributing to the greater accuracy of the generated article, thereby improving the overall accuracy of AI writing.
[0077] Furthermore, the aforementioned method, which requires users to download videos, use speech recognition software to convert them into text, correct errors in text recognition, and then input the text into AI writing software, as well as manually extracting images from the video to correspond to the text, is time-consuming and inefficient in generating articles. This can lead to delays, especially for time-sensitive news reports. In this embodiment, users do not need to download the original video; they only need to provide the target time period or the topic to be written. The automatic writing assistant and multimodal model can then automatically generate a draft article, reducing the steps of converting videos into text and manually extracting images from videos. This improves article generation efficiency and is particularly suitable for generating time-sensitive articles such as press releases.
[0078] In the embodiments of this application, such as Figure 2 As shown, article iteration and optimization can also be achieved through multiple rounds of human-computer dialogue. Each round of human-computer dialogue can correspond to one round of article generation or article writing. Specifically, for any writing round (e.g., the i-th round, i = 1, 2, 3, 4, ...), the article generated in that writing round can be output; if the writing round is the first round, then the article generated in that round is the initial draft of the article generated in step 104 above. For example, the article generated in the i-th writing round can be sent to the user's terminal device, and the user's terminal device can display the article.
[0079] In some embodiments, the layout format of the article generated by the device in the article generation method is a layout format in human-computer interaction mode. Accordingly, such as Figure 5 As shown in the image above, the article displayed on the user's terminal device also has the layout format of the human-computer interaction mode. For example, as... Figure 5 As shown in the image above, the layout format in the human-computer interaction mode includes: paragraphs of the main text arranged side-by-side with corresponding accompanying images, each paragraph numbered, and the name of the layout element preceding its specific content. For example, "Topic" is added before the article's theme, "Intelligent Connected Vehicles Boost Transportation Development." Numbering the paragraphs and labeling the layout elements with their names provides guidance for users to pinpoint the content to be modified when they request changes through human-computer interaction, facilitating precise modification requests.
[0080] For any writing round other than the first round (defined as round j, j = 2, 3, 4, ...), the user can use natural language to describe the modification requirements for the article generated in the previous round (round (j-1)). The method by which the user provides the modification requirements described in natural language to the device or equipment performing the article generation method is similar to the aforementioned method of providing article writing requirements described in natural language. For the second writing round, the article generated in the previous round is the initial draft of the article generated in step 104 above.
[0081] Furthermore, for the second writing round, the modification requirements for the first draft and the first draft can be input into the multimodal model together; the multimodal model can then generate the article for the second writing round based on the multimodal data, modification requirements, and the first draft.
[0082] Users can refine the generated article through multiple rounds of human-computer interaction, achieving multi-round article generation. The aforementioned embodiment only illustrates the article generation process for the second writing round; however, this writing process can be applied to any writing round. Specifically, for any writing round other than the first round of article generation (e.g., the j-th writing round, j = 2, 3, 4, ...), the user can use natural language to describe the modification requirements for the article output in the previous round (round (j-1)). Based on the j-th writing round other than the first round, the modification requirements for the article generated in round (j-1) can be described in natural language. Accordingly, as... Figure 2 As shown, for any current writing round (e.g., the j-th writing round, j = 2, 3, 4, ...), the device executing the article generation method (specifically, the automatic writing assistant running on that device) can obtain modification requirements for the article generated in the (j-1)-th round, described in natural language.
[0083] Furthermore, articles generated in the target historical writing rounds can be obtained. The target historical writing rounds can be all rounds preceding the current writing round. Alternatively, the target historical writing rounds can be determined by the current writing round and a pre-configured number of fusion rounds. The number of fusion rounds refers to the number of historical rounds supported by the j-th writing round. For example, if the number of fusion rounds is N, where N is a positive integer, then the j-th writing round supports the fusion of N historical writing rounds. Accordingly, if the number of writing rounds that have generated articles (i.e., historical writing rounds) is greater than or equal to N, the N historical writing rounds closest to the current writing round can be determined as the target historical writing rounds. If the number of writing rounds that have generated articles (i.e., historical writing rounds) is less than N, then the currently existing historical writing round is determined as the target historical writing round. For example, if the j-th writing round is the second round of writing, and N = 3, then the target historical round is the first round of writing. Accordingly, if the target historical writing rounds include the first round of writing, then the articles generated by the target historical writing rounds include the aforementioned draft articles.
[0084] Furthermore, such as Figure 2 As shown, the modification requirements for the article generated in the (j-1)th writing round and the articles generated in the target historical writing rounds can be input into the multimodal model. In this multimodal model, the article for the jth writing round can be generated based on the multimodal data extracted from the target video segment, the modification requirements, and the articles generated in the target historical writing rounds.
[0085] In this embodiment, by continuously interacting with users, modification requirements for the article can be obtained. Based on the obtained modification requirements and articles generated in the historical writing rounds, the article can be optimized, thereby improving the accuracy of parsing users' article writing requirements and the accuracy of the generated article. This helps to improve the quality of the generated article and meet the personalized needs of different users.
[0086] like Figure 5 As shown, for any writing round (e.g., the i-th writing round, i = 1, 2, 3, 4, ...), the articles generated in that writing round can be output. Among them, Figure 5 The above diagram illustrates an article generated in any writing round. For example, the article generated in the i-th writing round can be sent to the user's terminal device, which can then display the article. The article generated by the device using this article generation method is formatted in the human-computer interaction mode, and the article displayed on the user's terminal device is also formatted in the human-computer interaction mode. The device executing the article generation method can also exit the human-computer interaction mode in response to confirmation information regarding the article generated in the i-th writing round.
[0087] In some embodiments, when outputting the article generated in the i-th writing round, an article confirmation control can also be displayed on the user interface, providing a confirmation control for the article generated in the i-th writing round. If the user is satisfied with the article generated in the i-th writing round, the confirmation control can be triggered to confirm the final draft, such as by clicking the confirmation control. Accordingly, the device executing the article generation method can determine that it has received confirmation information for the article generated in the i-th writing round upon receiving a trigger operation for the confirmation control. Furthermore, in response to the confirmation information for the article generated in the i-th writing round, the device can exit the human-computer interaction mode.
[0088] In other embodiments, the device performing the article generation method may, after outputting the article generated in the i-th writing round, output a modification request message for the article generated in the i-th writing round in voice form. For example, "Do you have any further suggestions for the article? If no modifications are needed, please confirm." If the user is satisfied with the article generated in the i-th writing round and confirms that no further modifications are needed, such as... Figure 5As shown in the figure below, the feedback from the modification inquiry information for the article generated in the i-th writing round can reflect the intention to confirm, such as voice data like "Okay, very satisfied," "Confirm no further modifications," or "Okay, please finalize." Accordingly, the apparatus or device executing the article generation method can determine that it has received confirmation information for the article generated in the i-th writing round upon acquiring the voice data reflecting the intention to confirm based on the modification inquiry feedback. Furthermore, in response to the confirmation information for the article generated in the i-th writing round, the human-computer interaction mode can be exited.
[0089] For the device executing the article generation method, in response to the confirmation information for the article generated in the i-th writing round, the article generated in the i-th writing round can also be formatted according to a pre-set publishing version format to obtain a target article with a publishing version format. In the embodiments of this application, for ease of description and distinction, the formatting format in the aforementioned human-computer interaction mode is defined as the first formatting format; and the publishing version format is defined as the second formatting format. The second formatting format may be the same as or different from the first formatting format. For example, such as... Figure 6a As shown, the second formatting of the target article is the same as... Figure 5 The article above has the same initial layout format in the human-computer interaction mode; for example... Figure 6b As shown, the second formatting of the target article is the same as... Figure 5 The article in the image above has a different first layout format in the human-computer interaction mode. Preferably, the second layout format is different from the first layout format, so that users can intuitively distinguish the current mode and determine whether they are in human-computer interaction mode or completion mode.
[0090] Furthermore, the device executing the article generation method can output a target article with a second typesetting mode. For example, the target article with the second typesetting mode can be sent to the user's terminal device. The user's terminal device can then display the target article with the second typesetting format. This target article is the final draft.
[0091] like Figure 6a and Figure 6b As shown, the terminal device can display the target article. The display interface of the target article may include a save control and a publish control. Users can trigger the corresponding controls according to their needs. If the user triggers the save control, the terminal device can respond to the save trigger operation for the target article and save it. If the user triggers the publish control, the terminal device can respond to the publish trigger operation for the target article and publish it. For example, the target article can be published to a pre-set channel.
[0092] It is worth noting that the articles in this application embodiment can be of any genre or form. For example, the articles can be news articles, poems, WeChat official account articles, or website articles. Among them, news articles can also be called press releases. Figures 3-5 and Figure 6a and Figure 6b The illustrations used here are merely examples of press releases and are not intended to be limiting. The article generation method provided in this application is particularly suitable for generating time-sensitive articles such as news articles. The following is an illustrative description of the application of the article generation method provided in this application to press release generation.
[0093] Figure 7 This is a flowchart illustrating the press release generation method provided in an embodiment of this application. Figure 7 As shown, the press release generation method may include:
[0094] 701. Extract the target video clips corresponding to the topic to be written from the original videos that provide news materials.
[0095] 702. Obtain news writing requirements described in natural language.
[0096] 703. Extract multimodal data from the target video segment.
[0097] 704. Generate a first draft of the news based on multimodal data and news writing requirements using a multimodal model.
[0098] In this embodiment, the multimodal model uses multimodal data from the target video segment corresponding to the topic to be written, rather than the entire original video, to generate a draft news article. This allows the multimodal model to focus on the topic the user expects to write about (i.e., the topic to be written), improving the alignment between the generated news article and the user's expectations, thus increasing the accuracy of the generated news article. Furthermore, the multimodal model generates the draft news article based on the multimodal data of the target video segment. This multimodal data provides more reference information for the generated news article, enriching the dimensions of reference information for AI writing and making the news article generated by the multimodal model more accurate.
[0099] Furthermore, in traditional solutions, users need to download videos, use speech recognition software to convert the videos into text, correct any errors in the text recognition, and then input the text into AI writing software. Additionally, they need to manually extract images from the video to correspond to the text. This entire process is time-consuming, resulting in low news release generation efficiency and potentially delays in news reporting. In this embodiment, users do not need to download the original video. They only need to provide the target time period or the topic to be written. The automatic writing assistant and multimodal model can then automatically generate a draft article, reducing the steps of converting videos to text and manually extracting images from the video. This helps improve the efficiency of news release generation and ensures the timeliness of news reporting.
[0100] For specific implementation methods of steps 701-704, please refer to the foregoing. Figures 1-5 and Figure 6a and Figure 6b The relevant content will not be repeated here.
[0101] In this embodiment, the iteration and optimization of the press release can also be achieved through multiple rounds of human-computer dialogue. For any writing round other than the first round of press release generation (e.g., the j-th writing round, j = 2, 3, 4, ...), the user can use natural language to describe the modification requirements for the press release output in the previous round (round (j-1)). Based on the j-th writing round other than the first round, the modification requirements for the press release generated in round (j-1) can be described in natural language. Accordingly, for any current writing round (e.g., the j-th writing round, j = 2, 3, 4, ...), the device executing the press release generation method can obtain the modification requirements for the press release generated in round (j-1) described in natural language.
[0102] Furthermore, the press releases generated in the target historical writing rounds can be obtained. A description of the target historical writing rounds can be found in the relevant content of the foregoing embodiments, and will not be repeated here. Furthermore, the modification requirements for the press release generated in the (j-1)th writing round and the press releases generated in the target historical writing rounds can be input into the multimodal model together. In this multimodal model, the press release for the j-th writing round can be generated based on the multimodal data extracted from the target video segment, the modification requirements, and the press releases generated in the target historical writing rounds.
[0103] In this embodiment, by continuously interacting with users, modification requests for the press release can be obtained. Based on the obtained modification requests and the press releases generated in the previous writing rounds, the press releases can be optimized, thereby improving the accuracy of parsing users' press release writing requests and the accuracy of the generated press releases. This helps to improve the quality of the generated press releases and meet the personalized needs of different users.
[0104] For any given writing round (e.g., the i-th writing round, i = 1, 2, 3, 4, ...), the press release generated in that writing round can be output. For example, the press release generated in the i-th writing round can be sent to the user's terminal device, where the user's terminal device can display the press release.
[0105] Before the user confirms the final draft, the device executing the press release generation method is in human-computer interaction mode, and the press releases generated in each writing round are formatted according to the first layout format of the default human-computer interaction mode. That is, the press releases generated in each writing round have the first layout format of the human-computer interaction mode.
[0106] For the device executing the press release generation method, in response to confirmation information for the press release generated in the i-th writing round, it can exit the human-computer interaction mode and format the article generated in the i-th writing round according to a pre-set second layout format of the published version to obtain a target press release with the layout format of the published version. The target press release is the final draft. Furthermore, a target article with the second layout format can be output. For example, a target press release with the second layout format can be sent to a user's terminal device. The user's terminal device can display the target article with the second layout format. Of course, the user's terminal device can also save and / or publish the target press release with the second layout format.
[0107] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 and 102 can be device A; or the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and so on.
[0108] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.
[0109] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause one or more processors to perform the steps in the above-described article generation method and / or press release generation method.
[0110] This application also provides a computer program product, including a computer program that, when executed by one or more processors, causes the one or more processors to perform the steps in the above-described article generation method and / or press release generation method.
[0111] In the embodiments of this application, the specific implementation form of the computer program product is not limited. In some embodiments, the computer program product may be implemented as an application (APP), a mini-program, a computer-side client, a program module, a plug-in, an installation package, a software development kit (SDK), an image file of an optical disc (such as an ISO file), a system tool assistant, or software in the form of Software as a Service (SaaS), etc., but is not limited to these.
[0112] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device includes a memory 80a and a processor 80b. The memory 80a is used to store computer programs.
[0113] Processor 80b is coupled to memory 80a and is used to execute computer programs to perform the steps in the article generation method and / or news release generation method provided in the foregoing embodiments. Specific implementation details of each step can be found in the relevant descriptions of the foregoing embodiments, and will not be repeated here.
[0114] In some alternative implementations, such as Figure 8 As shown, the electronic device may also include optional components such as a communication component 80c, a power supply component 80d, a display component 80e, and an audio component 80f. Figure 8 The diagram only shows some components and does not mean that the electronic device must contain them. Figure 8 The inclusion of all components does not imply that an electronic device can only include... Figure 8 The components shown.
[0115] in addition, Figure 8 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the form factor of the electronic device. The electronic device in this embodiment can be a desktop computer, laptop computer, mobile phone, or IoT device; it can also be a traditional server, cloud server, or server cluster, or other server equipment.
[0116] In this embodiment, the memory is used to store computer programs and can be configured to store various other data to support operation on its host device. The processor can execute the computer programs stored in the memory to implement corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Electrically Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0117] In the embodiments of this application, the processor can be any hardware processing device capable of executing the above-described method logic. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a microcontroller unit (MCU); it can also be a field-programmable gate array (FPGA), a programmable array logic (PAL), a general array logic (GAL), a complex programmable logic device (CPLD), or other programmable devices; or it can be an advanced RISC machine (ARM) or a system on chip (SoC), etc., but is not limited thereto.
[0118] In this embodiment, the communication component is configured to facilitate wired or wireless communication between its host device and other devices. The device housing the communication component can access wireless networks based on communication standards, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In another exemplary embodiment, the communication component may also be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), or other technologies.
[0119] In embodiments of this application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.
[0120] In this embodiment, a power supply component is configured to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.
[0121] In embodiments of this application, the audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. For example, in devices with voice interaction capabilities, voice interaction with the user can be achieved through the audio component.
[0122] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.
[0123] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.
[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0127] In a typical configuration, a computing device includes one or more processors (CPU, etc.), input / output interfaces, network interfaces, and memory.
[0128] Memory may include non-persistent storage in computer-readable media, such as random-access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0129] Computer storage media are readable storage media, also known as removable media. Removable and non-removable media can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.
[0130] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.
[0131] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. An article generation method characterized by, The method comprises the following steps: extracting a target video segment corresponding to a to-be-written topic from an original video providing writing materials; obtaining an article writing requirement described in natural language; extracting multi-modal data from the target video segment; generating a draft of the article by a multi-modal model according to the multi-modal data and the article writing requirement.
2. The method of claim 1, wherein, The step of extracting a target video segment corresponding to a to-be-written topic from an original video providing writing materials comprises the following steps: obtaining a target time period selected according to the to-be-written topic; extracting a video segment of the target time period from the original video as the target video segment.
3. The method of claim 2, wherein, The step of obtaining a target time period selected according to the to-be-written topic comprises the following steps: providing a time selection control for the original video to allow a user to select a time period according to the to-be-written topic; obtaining a time period selected based on the time selection control as the target time period.
4. The method of claim 1, wherein, The step of extracting multi-modal data from the target video segment comprises the following steps: extracting at least two modal data of image data, voice data, subtitle data and background sound data from the target video segment by the multi-modal model as the multi-modal data.
5. The method of claim 1, wherein, The step of generating a draft of the article by a multi-modal model according to the multi-modal data and the article writing requirement comprises the following steps: performing feature extraction on the multi-modal data in the multi-modal model to obtain a feature vector of the multi-modal data; performing vectorization processing on the article writing requirement to obtain a feature vector of the article writing requirement; performing feature fusion on the feature vector of the multi-modal data and the feature vector of the article writing requirement to obtain a target feature vector; generating the draft of the article according to the target feature vector.
6. The method of claim 5, wherein, The step of generating the draft of the article according to the target feature vector comprises the following steps: determining the text content of the draft of the article according to the target feature vector in the multi-modal model; determining a picture corresponding to the text content from the target video segment according to the text content; performing layout on the text content and the picture to obtain the draft of the article.
7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises the following steps: entering a human-computer interaction mode in response to a writing request; outputting inquiry information for the article writing requirement in the form of voice; The step of obtaining an article writing requirement described in natural language comprises the following steps: obtaining voice data for describing the article writing requirement based on feedback of the inquiry information as the article writing requirement described in natural language.
8. The method according to any one of claims 1 to 6, characterized in that, The method further comprises the following steps: for any writing round, outputting an article generated in the any writing round to allow a user to confirm or propose a modification requirement for the article generated in the any writing round; if the any writing round is a first round, the article generated in the any writing round is the draft of the article; obtaining a modification requirement for the article generated in the any writing round described in natural language; inputting the modification requirement and an article generated in a target historical writing round into the multi-modal model; generating an article of a next writing round of the any writing round in the multi-modal model according to the multi-modal data, the modification requirement and the article generated in the target historical writing round.
9. The method of claim 8, wherein, The output of the article generated in any writing round includes: sending the article generated in any writing round to the terminal device of the user, so that the terminal device displays the article generated in any writing round in a first layout format in a human-computer interaction mode; the article generated in any writing round has the first layout format; The method further includes: in response to the confirmation information for the article generated in any writing round, exiting the human-computer interaction mode, and laying out the article generated in any writing round in a second layout format to obtain a target article with the second layout format; sending the target article with the second layout format to the terminal device, so that the terminal device saves and / or publishes the target article with the second layout format.
10. A news article generation method characterized by comprising: including: extracting a target video segment corresponding to the to-be-written topic from the original video providing news materials; obtaining news writing requirements described in natural language; extracting multi-modal data from the target video segment; generating a news draft through a multi-modal model according to the multi-modal data and the news writing requirements.
11. The method of claim 10, wherein, Also includes: for outputting a news draft generated in any writing round; if the any writing round is the first writing round, the news draft generated in the any round is the news draft; obtaining modification requirements described in natural language for the news draft generated in the any round; inputting the modification requirements and the news draft generated in the target historical writing round into the multi-modal model, and generating a news draft in the current writing round according to the multi-modal data, the modification requirements, and the news draft generated in the target historical writing round in the multi-modal model.
12. An electronic device, comprising: including: a memory and a processor; wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps in the method of any one of claims 1-11.
13. A computer readable storage medium having stored thereon computer instructions, wherein: When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps in the method of any one of claims 1-11.
14. A computer program product, characterised in that, including a computer program, when the computer program is executed by one or more processors, the one or more processors are caused to perform the steps in the method of any one of claims 1-11.