Multimedia content generation methods, apparatus, devices, and media
By parsing and vectorizing multimedia content and establishing index relationships, the problem of artificial intelligence models ignoring text content in their answers is solved, resulting in more accurate and relevant responses.
Patent Information
- Application Number
- CN202310997018.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-08
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-08-08
AI Technical Summary
Existing artificial intelligence models tend to ignore the text content itself when answering user questions, focusing more on training sample information, resulting in low accuracy and poor performance of the answers.
By parsing multimedia content, generating vector data and establishing index relationships, the target vector data is retrieved from the vector database using the index relationships, and then input into an artificial intelligence model to provide an answer, ensuring that the answer is relevant to the multimedia content.
This improves the accuracy and relevance of the answers, ensuring a strong connection between the answers and the multimedia content.
Smart Images

Figure CN117076602B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more specifically, to a method, apparatus, device, and medium for generating multimedia content. Background Technology
[0002] With the rapid development of artificial intelligence (AI) technology, various AI models are being widely applied and playing an increasingly important role in various fields. While conventional AI models can learn from text content and engage in dialogue with users based on the learned information, they are trained on a large number of training samples before use. Therefore, even after learning from the text content, the model's responses to user questions may ignore the text content itself and focus more on the information learned from the training samples. This can lead to lower accuracy and poor performance in responses, thus affecting the effectiveness of the AI model. Summary of the Invention
[0003] This disclosure provides at least one method, apparatus, device, and medium for generating multimedia content.
[0004] In a first aspect, embodiments of this disclosure provide a content generation method based on multimedia content, including:
[0005] The acquired multimedia content is parsed to obtain multiple text information items.
[0006] Based on the degree of textual association between the content text information, the content text information is stored in a vector database in the form of vector data; the vector database has an index relationship between each vector data related to the degree of textual association.
[0007] In response to receiving input query information, based on the index relationship between vector data in the vector database, the system searches the vector database for target vector data that match the query information.
[0008] The target content text information corresponding to the target vector data and the question information are input into the artificial intelligence model to obtain the answer result associated with the multimedia content.
[0009] In one possible implementation, after obtaining multiple pieces of content text information, the method further includes:
[0010] By inputting multiple pieces of the content text information into the artificial intelligence model, a summary report information corresponding to the multimedia content is obtained;
[0011] The summary report information is sent to the client, which then displays the summary report information upon receiving it.
[0012] In one possible implementation, based on the index relationships between vector data in the vector database, searching for target vector data that match the query information from the vector database includes:
[0013] Based on the semantic features of the question information, at least one target query term corresponding to the question information is generated; different target query terms have different query semantics.
[0014] Based on each of the target query terms and the index relationship between the vector data in the vector database, the target vector data that matches the query information are determined.
[0015] In one possible implementation, the target content text information corresponding to the target vector data and the question information are input into an artificial intelligence model to obtain an answer result associated with the multimedia content, including:
[0016] The target content text information and the question information are input into the artificial intelligence model, and the artificial intelligence model is used to generate the answer result associated with the multimedia content according to the corresponding content generation rules.
[0017] In one possible implementation, the target content text information corresponding to the target vector data and the question information are input into an artificial intelligence model to obtain an answer result associated with the multimedia content, including:
[0018] Obtain historical dialogue information for each round prior to the question time in the question information; the historical dialogue information is dialogue information related to the multimedia content;
[0019] The historical dialogue information from each round, the target content text information, and the question information are input into the artificial intelligence model to obtain the answer results associated with the multimedia content.
[0020] In one possible implementation, the multimedia content includes at least one of text content in any text format and video content in any video format;
[0021] The multimedia content can be obtained using at least one of the following methods:
[0022] The system retrieves local text or video content uploaded by the client, and obtains the text or video content from the storage location corresponding to the uploaded content link.
[0023] In one possible implementation, when the multimedia content includes video content, the content text information includes image text information corresponding to each video frame image;
[0024] Inputting multiple pieces of the content text information into the artificial intelligence model yields a summary report information corresponding to the multimedia content, including:
[0025] By inputting the text information of each image into the artificial intelligence model, key frame information, keyword information and video content summary information of the video content are obtained;
[0026] Based on the keyframe information, determine the keyframe images corresponding to the video content;
[0027] Based on the keyframe images, the keyword information, and the video content summary information, a summary report information corresponding to the video content is generated.
[0028] Secondly, embodiments of this disclosure also provide a content generation apparatus based on multimedia content, comprising:
[0029] The parsing module is used to parse the acquired multimedia content and obtain multiple content text information.
[0030] The storage module is used to store the content text information in the form of vector data into a vector database according to the degree of text correlation between the content text information; the vector database has an index relationship between each vector data related to the degree of text correlation.
[0031] The search module is used to respond to received input query information and, based on the index relationship between vector data in the vector database, search for target vector data that match the query information in the vector database.
[0032] The output module is used to input the target content text information corresponding to the target vector data and the question information into the artificial intelligence model to obtain the answer result associated with the multimedia content.
[0033] Thirdly, an optional implementation of this disclosure also provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.
[0034] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.
[0035] For a description of the effects of the aforementioned multimedia content generation apparatus, computer equipment, and computer-readable storage medium, please refer to the description of the multimedia content generation method above; it will not be repeated here.
[0036] The multimedia content generation method, apparatus, device, and medium provided in this disclosure, through content parsing of multimedia content, can determine multiple fragmented text information corresponding to the multimedia content. Based on the degree of textual correlation between the textual information, the textual information is stored in a vector database as vector data. This not only enables lightweight storage of each textual information using vector data but also allows for the representation of textual correlation using index relationships, improving the relevance between the stored vector data. Upon receiving a question about the multimedia content, the target vector data with a correlation to the question can be quickly and conveniently retrieved from the vector database using index relationships. Finally, by inputting the target content text information corresponding to the target vector data and the question information into an artificial intelligence model, the model can summarize and generalize the target content text information, outputting an accurate answer that matches the question while ensuring a strong correlation between the answer and the multimedia content, thus guaranteeing the effectiveness of the answer.
[0037] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0038] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.
[0039] Figure 1 A flowchart of a multimedia content generation method provided by an embodiment of this disclosure is shown;
[0040] Figure 2This illustration shows a process for displaying summary report information provided by an embodiment of the present disclosure;
[0041] Figure 3 This illustration shows another process for displaying summary report information provided by an embodiment of the present disclosure;
[0042] Figure 4 This illustration shows a process for generating an answer result provided by an embodiment of the present disclosure;
[0043] Figure 5 This diagram illustrates a multimedia content generation apparatus provided in an embodiment of the present disclosure.
[0044] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.
[0046] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.
[0047] In this article, "multiple or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0048] Research has found that after uploading texts, books, and other materials to an AI model for learning, the model can learn content features. However, when a user initiates a dialogue question about the uploaded content, the AI model's response often only focuses on a small portion of the uploaded content. Instead, the majority of the response is based on the large-scale samples that the model relied on during training. This results in low accuracy and poor performance of the AI model's responses.
[0049] Based on the above research, this disclosure provides a method, apparatus, device, and medium for generating multimedia content. By parsing the multimedia content, multiple fragmented textual information pieces corresponding to the multimedia content can be identified. Based on the degree of textual correlation between the textual information pieces, the content textual information is stored in a vector database as vector data. This not only enables lightweight storage of each textual information piece using vector data but also allows for the representation of textual correlation using index relationships, improving the relevance between the stored vector data. Upon receiving a question about the multimedia content, the target vector data with a correlation to the question can be quickly and conveniently retrieved from the vector database using index relationships. Finally, by inputting the target content textual information corresponding to the target vector data and the question information into an artificial intelligence model, the model can summarize and generalize the target content textual information, outputting accurate answers that match the question information while ensuring a strong correlation between the answer and the multimedia content, thus guaranteeing the effectiveness of the answer.
[0050] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.
[0051] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0052] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0053] To facilitate understanding of this embodiment, a detailed description of a multimedia content generation method disclosed in this disclosure will be provided first. The execution entity of the multimedia content generation method provided in this disclosure is generally a terminal device or other processing device with certain computing capabilities. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, personal digital assistant (PDA), handheld device, computer device, etc. In some possible implementations, the multimedia content generation method can be implemented by a processor calling computer-readable instructions stored in memory.
[0054] The following describes the content generation method based on multimedia content provided in this disclosure embodiment, taking the server as the executing entity as an example.
[0055] like Figure 1 The flowchart shown is a method for generating multimedia content based on an embodiment of this disclosure, which may include the following steps:
[0056] S101: Perform content parsing on the acquired multimedia content to obtain multiple content text information.
[0057] Here, multimedia content can be any type of content and can be in any format, such as image content in any image format, video content in any video format, book content, document content in any format, etc.
[0058] Content text information can be text information parsed from multimedia content. These individual pieces of content text information can be segmented and broken down, and different pieces of text information can have different lengths and semantics. Multiple pieces of content text information can represent multimedia content, and the number of pieces of content text information can be determined based on the amount of multimedia content resources and the degree of content repetition.
[0059] In practice, a smart reading tool can be pre-developed and installed on any client. When a user stores reading requests, the client can respond to the user's trigger action on the installed smart reading tool, displaying the tool's dialog page. Then, the user can use the client to upload the multimedia content they want to read to the server. The server can then obtain the uploaded multimedia content and call the corresponding parsing function of the parsing plugin to parse and extract the content, obtaining multiple pieces of text information.
[0060] It should be noted that the content generation method based on multimedia content provided in this disclosure is applicable to multimedia content reading scenarios. After a user uploads multimedia content, an artificial intelligence model can be used to parse the multimedia content according to the content generation method based on multimedia content provided in this disclosure, and the parsed content text information can be stored in a vector database. Then, after receiving a user's question about the multimedia content, the answer result based on the multimedia content can be generated using the target content text information found in the vector database.
[0061] In one embodiment, the multimedia content may include at least one of text content in any text format and video content in any video format. Here, the text format may be, for example, PDF, doc, txt, csv, xlsx, etc., and the video format may be, for example, mp4, AVI, MPEG, etc. In specific implementations, any possible text or video format may fall within the protection scope of this disclosure.
[0062] Regarding S101 above, the multimedia content can be obtained using at least one of the following two methods:
[0063] Method 1: Obtain local text content or local video content uploaded by the client.
[0064] Here, local text content refers to text content of any text format stored locally on the client, and local video content refers to video content of any video format stored locally on the client.
[0065] For example, the server can obtain any local text content or any local video content uploaded by the client.
[0066] Method 2: Obtain the text or video content from the storage location corresponding to the uploaded content link.
[0067] Here, content links can be URL links corresponding to any webpage or location links corresponding to any storage location.
[0068] For example, the server can obtain the link to the content uploaded by the client, and then retrieve the multimedia content from the storage location indicated by the content link.
[0069] Optionally, since the acquired multimedia content may be text or video content, and may have any format, to improve the accuracy of the parsed text information, different specialized parsing plugins can be pre-set for text content of various formats and video content of various formats. After acquiring the multimedia content, the server can call the corresponding specialized parsing plugin according to the format and type of the multimedia content (i.e., text type or video type), and use the parsing function of the specialized parsing plugin to parse and extract the multimedia content, thereby obtaining multiple pieces of text information.
[0070] This allows for uploading multimedia content in various formats using multiple methods, thus increasing the diversity of multimedia content that can be processed.
[0071] S102: Based on the degree of textual association between content text information, store the content text information in the form of vector data in a vector database; the vector data in the vector database have an index relationship related to the degree of textual association.
[0072] Here, textual relevance is used to characterize the degree of informational connection between two pieces of content text. Each piece of content text can have a degree of textual relevance with every other piece of content text. The degree of textual relevance can be determined based on the semantic features, coherence, and relevance between the two pieces of content text. Coherence between content text is used to characterize whether the text formed by combining two pieces of content text is continuous and fluent, while relevance between content text is used to characterize the frequency of use of the text formed by combining two pieces of content text. For example, regarding the content text information "rain," "umbrella," and "outing," the coherence and relevance between "rain" and "umbrella" are better than the coherence and relevance between "rain" and "outing."
[0073] A vector database can store at least the vector data corresponding to each piece of text information.
[0074] Specifically, after obtaining the various textual information pieces, vector data corresponding to each piece of textual information can be generated, and then stored in a vector database. Simultaneously, an index relationship can be established between the vector data pieces corresponding to each piece of textual information based on the degree of textual association between them. Here, for each vector data piece, a corresponding index relationship can exist. This index relationship can be determined by sorting the textual information corresponding to that vector data piece in descending order of the degree of textual association with other textual information pieces. Alternatively, the index relationship can be constructed based on the degree of textual association between each piece of textual information piece and a preset threshold. For example, if the degree of textual association between textual information 1 and textual information 2 corresponding to vector data 1 is greater than the preset threshold, an index relationship can be established between vector data 1 and vector data 2 corresponding to textual information 2; conversely, if the degree of textual association between textual information 1 and textual information 2 is not greater than the preset threshold, there will be no index relationship between vector data 1 and vector data 2.
[0075] In practice, after parsing the content text information, the degree of textual correlation between any two pieces of content text can be determined, and vector data corresponding to each piece of content text can be generated. Then, these vector data can be stored in a pre-established vector database, and an index relationship between the vector data can be constructed based on the degree of textual correlation between the content text information corresponding to each vector data. In this way, by storing vector data in a vector database, the scattered storage of multiple pieces of content text corresponding to multimedia content can be achieved.
[0076] Optionally, the parsed text information can be stored in a vector database or in a pre-created text database.
[0077] S103: In response to receiving the input query information, based on the index relationship between the vector data in the vector database, search for each target vector data that matches the query information in the vector database.
[0078] Here, the question information can be questions raised by the user regarding the uploaded multimedia content. For example, the question could be about the meaning of a word appearing in the multimedia content, or it could be about the content structure of the multimedia content. The target vector data can be the vector data found that can be used to answer the question information, and it belongs to the vector data corresponding to the user-uploaded multimedia content.
[0079] In practice, after storing the content text information in the vector database, the server can send a notification message to the client indicating that parsing is complete. For example, the notification message could be "Content parsed successfully, you can ask a question now." The client can then display this notification message on the chat page. The user can then ask a question about the uploaded multimedia content by entering the question in the corresponding input box on the chat page, and the client can send the question to the server. Upon receiving the question, the server can calculate the matching degree between the question and each vector data point. Using the matching degree and the index relationships between the vector data points, it can retrieve the target vector data points that match the question from the vector database. For example, if vector data 1 is found using the matching degree, vector data 3 and vector data 4, indicated by the index relationship of vector data 1, can be used together as the target vector data.
[0080] In one embodiment, S103 can also be implemented according to the following steps:
[0081] S103-1: Based on the semantic features of the query information, generate at least one target query term corresponding to the query information; different target query terms have different query semantics.
[0082] Here, a target query term can be called a query, which is obtained after semantic analysis of the query information. Different query semantics correspond to different query directions and query targets.
[0083] In practice, after obtaining the question information, semantic recognition and analysis can be performed on the question information to determine its semantic features. Then, based on the semantic features, the various query semantics present in the question information can be determined, and the target query terms corresponding to each query semantic can be generated.
[0084] S103-2: Based on each target query term and the index relationship between each vector data in the vector database, determine each target vector data that matches the query information.
[0085] In practice, for each target query term, the target vector data matching each target query term can be determined based on the relevance between the target query term and the vector data in the vector database, as well as the indexing relationships between the vector data. Then, the target vector data matching each target query term can be aggregated to obtain the target vector data matching the query information.
[0086] In this way, by performing semantic analysis on the query information, the query information can be abstracted into a query. Using the query to retrieve target vector data from the vector database can improve the comprehensiveness and accuracy of the retrieved vector data.
[0087] In one embodiment, after obtaining multiple content text information, the multiple content text information can be further summarized. Specifically, the multiple content text information can be input into an artificial intelligence model to obtain summary report information corresponding to the multimedia content; the summary report information is then sent to the client, which displays the summary report information upon receiving it.
[0088] Here, the summary report information is information summarized using artificial intelligence models, which may include, but is not limited to, a content introduction, text outline, text summary, text keywords, key text content, frequency of keyword occurrences, location of key content, and segmented summaries.
[0089] In practice, after obtaining multiple pieces of textual information, an artificial intelligence (AI) model can be invoked. The AI model then identifies and summarizes the multiple pieces of textual information to generate a summary report of the multimedia content. Specifically, for keywords, filtering can be applied to specific types of words, such as common modal particles and auxiliary words. The server can then send the summary report to the client, which can display the received summary report as a notification on the chat page.
[0090] like Figure 2 The diagram shown is a schematic representation of a process for displaying summary report information according to an embodiment of this disclosure. Figure 2 In the context of 'a', the multimedia content displayed on the dialog page is a text content 1 in txt format. After uploading text content 1, the dialog page can display parsing prompts (i.e., ... Figure 2 The document in section 'a' states, "I will parse the file and extract summary report information such as 'text content summary, text abstract, key information, and segmented summary.' I can also answer any questions about the file; feel free to ask!" and provides progress indicators (i.e., ...). Figure 2 (In the "file parsing in a" section). After the server obtains the summary report information using an artificial intelligence model, it can send the summary report information back to the client, which will then display it as follows: Figure 2 The dialog page shown as 'b' displays a summary report corresponding to text content 1. This summary report includes an introduction, segmented summaries (i.e., key points 1 and 2), multiple keywords (i.e., keywords 1-3), and their corresponding frequency and key content (i.e.,...). Figure 2 (Key content 1 and 2 in b). The dialog page also displays a dialog input box, which supports both text and voice input. Users can use the dialog input box to ask questions about content text 1.
[0091] In addition, it can also respond to a trigger operation on text content 1 in the dialog page to display the text content details page corresponding to text content 1, thus enabling the display of the source text.
[0092] In one embodiment, when the multimedia content is video content, the content text information may include image text information corresponding to each video frame. That is, when the multimedia content is video content, text recognition processing can be performed on each video frame to obtain image text information corresponding to each video frame, and this image text information can be used as the content text information corresponding to the video content.
[0093] The steps for generating the summary report information can be implemented as follows:
[0094] Step 1: Input the text information of each image into the artificial intelligence model to obtain keyframe information, keyword information and video content summary information of the video content.
[0095] Here, keyframe information is used to indicate each keyframe image in the video content, specifically a portion of the image text information from the input image text information. Keyword information is used to indicate each keyword in the video content; video content summary information can include a video content introduction, video segment summaries, main content summary, video arguments, etc.
[0096] In practical implementation, when the multimedia content is video, video parsing can be performed first to obtain the image and text information corresponding to each frame of the video. Then, an artificial intelligence model can be invoked, and the image and text information can be input into the model. The AI model then identifies and summarizes the image and text information to obtain keyframe information, keyword information, and summary information of the video content. When extracting key keywords, specific types of words can be filtered, such as common modal particles, auxiliary words, prepositions, and locative words.
[0097] Step 2: Based on the keyframe information, determine the keyframe images corresponding to the video content.
[0098] In practice, an artificial intelligence model can be used to generate a keyframe image that matches each keyframe information based on the keyframe information. Alternatively, the artificial intelligence model can be used to call an image search function to find the keyframe image corresponding to each keyframe information in the video content.
[0099] Step 3: Generate a summary report of the video content based on the keyframe images, keyword information, and video content summary information.
[0100] In practice, keyframe images, keyword information, and video content summary information can be structured and combined to obtain summary report information corresponding to the video content.
[0101] Optionally, after acquiring the user-uploaded video content, the keyframe extraction function of the video analysis plugin can be used to extract keyframe images from the video content. Then, after inputting the image and text information into the artificial intelligence model, keyword information and video content summary information can be obtained. Then, a summary report can be generated based on the keyword information, video content summary information, and keyframe images extracted using the keyframe extraction function. Specifically, keyframe images can be extracted by performing differential processing on every two video frames to obtain the average pixel intensity between the two frames. The average pixel intensity can be used to measure the degree of change in the content of the two video frames. Then, video frames with an average pixel intensity greater than a preset intensity can be used as the extracted keyframe images. In this way, video frames with a large degree of change in content can be extracted.
[0102] like Figure 3 The diagram shown illustrates another process for displaying summary report information provided in this embodiment of the disclosure. Figure 3 In the context of 'c', the multimedia content displayed on the dialog page is an MP4 video content 1. After uploading video content 1, the dialog page can display parsing prompts (i.e.,...). Figure 3 The text in section c states, "I will analyze the video and extract summary report information such as 'video summary, segment summary, and key information.' I can also answer any questions about the video; feel free to ask!" and includes analysis progress prompts (i.e.,...). Figure 3 (In the "Video parsing in c" section). After the server uses an artificial intelligence model to obtain the summary report information, it can send the summary report information back to the client, which will then display it as follows: Figure 3 The dialog page shown as 'd' displays a summary report of video content 1. This summary report includes an introduction, segmented summaries (i.e., key points 3 and 4), multiple keywords (i.e., keywords 4-6), their corresponding frequency of occurrence, and keyframe images (i.e., keyframe images 1-3). The dialog page also displays a dialog input box, which supports both text and voice input, allowing users to ask questions about video content 1.
[0103] Furthermore, in response to a trigger action on video content 1 in the dialog page, the video content details page corresponding to video content 1 can be displayed, thus enabling the display of the source video. Moreover, while displaying the source video, the hidden playback control on the page can be triggered to enter an immersive state, achieving hidden playback of the source video.
[0104] S104: Input the target content text information corresponding to the target vector data and the question information into the artificial intelligence model to obtain the answer results associated with the multimedia content.
[0105] Here, the answer result is the result used to respond to the question information, and this result is summarized by the artificial intelligence model. Since the answer result is generated based on the target content text information corresponding to the target vector data, the answer result will have a strong correlation with the multimedia content uploaded by the user. The target content text information is the content text information used to generate the target vector data.
[0106] In practice, after determining each target vector data, the target content text information corresponding to each target vector data can be retrieved. If the target content text information is stored in a vector database, it can be directly retrieved from the vector database; if it is stored in a text database, it can be retrieved from the text database. For each target vector data, it can carry the location information of the target content text information corresponding to that target vector data, allowing the target content text information to be retrieved directly from its designated storage location.
[0107] After obtaining the text information of each target content, an artificial intelligence model can be invoked. The question information and the text information of each target content can be input into the artificial intelligence model. The artificial intelligence model can then identify and process the question information and summarize the text information of each target content, outputting an answer result associated with the multimedia content. This answer result is used to answer the question information.
[0108] In one embodiment, to further improve the accuracy and reasonableness of the generated answer, the above-mentioned S104 can also be implemented according to the following steps:
[0109] The target content text information and question information are input into the artificial intelligence model, which then generates an answer result associated with the multimedia content according to the corresponding content generation rules.
[0110] Here, the content generation rule information can be pre-set rules specifically for the artificial intelligence model within the reading scenario corresponding to the intelligent reading tool. This rule information assists the artificial intelligence model in generating answer results. For example, the content generation rule information may include generation topic information and generation constraint information. Specifically, the generation topic information may be a topic that tends to generate content based on multimedia content, and the generation constraint information may include format constraints, word count constraints, content structure constraints, and symbol usage constraints for the answer results.
[0111] In practice, after receiving a question, content generation rules matching the corresponding scenario (i.e., reading scenario) can be determined. Then, the target content text, content generation rules, and question are input into the AI model. The content generation rules constrain the AI model, resulting in an answer to the question. This answer is strongly correlated with the multimedia content corresponding to the target text and conforms to the generation rules indicated by the content generation rules, thus better meeting the user's question needs and achieving higher accuracy.
[0112] In another embodiment, to improve dialogue relevance, if there are historical dialogues on the same multimedia content prior to the current round of dialogue, the above S104 can also be implemented according to the following steps:
[0113] S104-1: Obtain historical dialogue information for each round prior to the question time in the question information; historical dialogue information refers to dialogue information related to multimedia content.
[0114] Here, a question and an answer to that question can be called a round of dialogue information. Historical dialogue information can be the dialogue information that occurred before the dialogue information corresponding to the current round of dialogue. Both historical dialogue information and current round of dialogue information are dialogue information related to multimedia content uploaded by the user.
[0115] The question time is the time when the question information is entered in the current round of dialogue, and the dialogue time can be the time when the historical question information is entered in each round of historical dialogue.
[0116] In practice, after obtaining the question information for the multimedia content, it can be determined whether there is historical dialogue information for that multimedia content based on the question time and the corresponding dialogue time in the historical dialogue information. If so, the historical dialogue information for each round before the question time can be obtained, and then S104-2 below can be executed. If not, the target content text information, content generation rule information, and question information can be directly input into the artificial intelligence model to obtain the answer result.
[0117] S104-2: Input the historical dialogue information from each round, the target content text information, and the question information into the artificial intelligence model to obtain the answer results associated with the multimedia content.
[0118] In practice, after obtaining the historical dialogue information of each round for multimedia content, the historical dialogue information, target content text information and question information can be input into the artificial intelligence model together to intervene in the generated answer results using the historical dialogue information, so as to obtain the answer results associated with the multimedia content. The answer results obtained here are strongly correlated with both the historical dialogue information and the target content text information.
[0119] Optionally, after obtaining the historical dialogue information for each round regarding the multimedia content, the historical dialogue information, target content text information, question information, and content generation rule information can be input together into the artificial intelligence model to obtain the answer results associated with the multimedia content. This not only allows for content constraints on the generated answer results but also provides information on the relevance of the generated answer results to the historical dialogue information and multimedia content, thereby improving the dialogue effect.
[0120] like Figure 4The diagram illustrates a process for generating an answer result according to an embodiment of this disclosure. The server receives multimedia content uploaded by the client. This multimedia content can be in any format, such as Excel, PPT, PDF, or MP4. The server then parses the data to obtain multiple text messages corresponding to the multimedia content. These text messages are stored in a vector database as vector data. Simultaneously, the multiple text messages can be input into an artificial intelligence model, which summarizes and generates a summary report of the multimedia content. This summary report is then sent back to the client, allowing the client to display it on the chat page. The user can then ask a question about the multimedia content on the client's chat page. The client receives the user's input and sends it to the server. The server abstracts the question into a target query term and, based on the index relationships between vector data in the vector database, determines the target vector data corresponding to the target query term. This allows the server to retrieve the target vector data matching the question from the vector database. Then, the target content text information corresponding to each target vector data can be determined. This target content text information and the question information are input into the artificial intelligence model, which then performs inductive summarization to obtain the answer results associated with the multimedia content. Optionally, the target content text information, question information, historical dialogue information, and content generation rule information can also be input into the artificial intelligence model together to obtain the answer results associated with the multimedia content. Finally, the server can feed back the answer results to the client, allowing the client to display the answer results to the user, thus completing one round of dialogue. About Figure 4 The specific implementation process of each step can be referred to the above embodiments, and will not be repeated here.
[0121] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0122] Based on the same inventive concept, this disclosure also provides a multimedia content generation device corresponding to the multimedia content generation method. Since the principle of the device in this disclosure is similar to the multimedia content generation method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0123] like Figure 5 The diagram shown is a schematic representation of a multimedia content generation apparatus provided in an embodiment of this disclosure, comprising:
[0124] The parsing module 501 is used to parse the acquired multimedia content to obtain multiple content text information.
[0125] Storage module 502 is used to store the content text information in the form of vector data into a vector database according to the degree of text association between the content text information; the vector database has an index relationship between each vector data related to the degree of text association;
[0126] The lookup module 503 is used to respond to the received input query information and, based on the index relationship between the vector data in the vector database, search for each target vector data that matches the query information in the vector database.
[0127] The output module 504 is used to input the target content text information corresponding to the target vector data and the question information into the artificial intelligence model to obtain the answer result associated with the multimedia content.
[0128] In one possible implementation, the device further includes:
[0129] After receiving multiple text messages, the sending module 505 is also used for:
[0130] By inputting multiple pieces of the content text information into the artificial intelligence model, a summary report information corresponding to the multimedia content is obtained;
[0131] The summary report information is sent to the client, which then displays the summary report information upon receiving it.
[0132] In one possible implementation, the search module 503, when searching for target vector data matching the query information from the vector database based on the index relationships between the vector data in the vector database, is configured to:
[0133] Based on the semantic features of the question information, at least one target query term corresponding to the question information is generated; different target query terms have different query semantics.
[0134] Based on each of the target query terms and the index relationship between the vector data in the vector database, the target vector data that matches the query information are determined.
[0135] In one possible implementation, the output module 504, when inputting the target content text information corresponding to the target vector data and the question information into the artificial intelligence model to obtain an answer result associated with the multimedia content, is used to:
[0136] The target content text information and the question information are input into the artificial intelligence model, and the artificial intelligence model is used to generate the answer result associated with the multimedia content according to the corresponding content generation rules.
[0137] In one possible implementation, the output module 504, when inputting the target content text information corresponding to the target vector data and the question information into the artificial intelligence model to obtain an answer result associated with the multimedia content, is used to:
[0138] Obtain historical dialogue information for each round prior to the question time in the question information; the historical dialogue information is dialogue information related to the multimedia content;
[0139] The historical dialogue information from each round, the target content text information, and the question information are input into the artificial intelligence model to obtain the answer results associated with the multimedia content.
[0140] In one possible implementation, the multimedia content includes at least one of text content in any text format and video content in any video format;
[0141] The device further includes an acquisition module 506, configured to acquire the multimedia content using at least one of the following methods:
[0142] The system retrieves local text or video content uploaded by the client, and obtains the text or video content from the storage location corresponding to the uploaded content link.
[0143] In one possible implementation, when the multimedia content includes video content, the content text information includes image text information corresponding to each video frame image;
[0144] The sending module 505, when inputting multiple pieces of the content text information into the artificial intelligence model to obtain summary report information corresponding to the multimedia content, is used to:
[0145] By inputting the text information of each image into the artificial intelligence model, key frame information, keyword information and video content summary information of the video content are obtained;
[0146] Based on the keyframe information, determine the keyframe images corresponding to the video content;
[0147] Based on the keyframe images, the keyword information, and the video content summary information, a summary report information corresponding to the video content is generated.
[0148] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0149] Based on the same technical concept, embodiments of this application also provide a computer device. (Refer to...) Figure 6 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application, comprising:
[0150] The system comprises a processor 601, a memory 602, and a bus 603. The memory 602 stores machine-readable instructions executable by the processor 601. The processor 601 executes these machine-readable instructions, performing the following steps: S101: Parsing the acquired multimedia content to obtain multiple content text information; S102: Storing the content text information in the form of vector data in a vector database based on the degree of textual correlation between the content text information; the vector data in the vector database have an index relationship related to the degree of textual correlation; S103: Responding to received input question information, searching the vector database for target vector data matching the question information based on the index relationship between the vector data; and S104: Inputting the target content text information corresponding to the target vector data and the question information into an artificial intelligence model to obtain an answer result associated with the multimedia content.
[0151] The aforementioned memory 602 includes a main memory 6021 and an external memory 6022. The main memory 6021, also known as internal memory, is used to temporarily store the computational data in the processor 601, as well as the data exchanged with external memory such as a hard disk 6022. The processor 601 exchanges data with the external memory 6022 through the main memory 6021. When the computer device is running, the processor 601 and the memory 602 communicate through the bus 603, so that the processor 601 executes the execution instructions mentioned in the above method embodiments.
[0152] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the multimedia content generation method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0153] The computer program product of the multimedia content generation method provided in this disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the multimedia content generation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0154] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0157] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0158] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0159] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.
[0160] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.
Claims
1. A content generation method based on multimedia content, comprising: In response to receiving multimedia content uploaded by the client, the multimedia content is parsed by calling a parsing plugin corresponding to the format of the multimedia content to obtain multiple content text information, wherein the multimedia content includes at least one of text content in any text format and video content in any video format; Based on the degree of textual association between the multiple content text information, the multiple content text information are stored in a vector database in the form of vector data; the vector database has an index relationship between each vector data related to the degree of textual association; In response to receiving a query related to the multimedia content, based on the index relationship between vector data in the vector database, the system searches the vector database for target vector data that match the query; and The target content text information corresponding to each target vector data and the question information are input into the artificial intelligence model to obtain the answer result generated by the artificial intelligence model and associated with the multimedia content. The step of responding to receiving a question related to the multimedia content, and searching for target vector data matching the question information in the vector database based on the index relationship between vector data in the vector database, includes: Based on the matching degree between the question information and each vector data in the vector database, find the first vector data that matches the question information; Based on the index relationship between the vector data in the vector database, find the second vector data that has an index relationship with the first vector data; The target vector data that match the question information includes the first vector data and the second vector data.
2. The method according to claim 1, wherein, After obtaining multiple pieces of text information, the method further includes: By inputting multiple pieces of the content text information into the artificial intelligence model, a summary report information corresponding to the multimedia content is obtained; The summary report information is sent to the client, which then displays the summary report information upon receiving it.
3. The method according to claim 1, wherein, The step of responding to receiving a query related to the multimedia content, and searching for target vector data matching the query based on the index relationship between vector data in the vector database, further includes: Based on the semantic features of the question information, at least one target query term corresponding to the question information is generated; different target query terms have different query semantics. Based on each of the target query terms and the index relationship between the vector data in the vector database, the target vector data that matches the query information are determined.
4. The method according to claim 1, wherein, The step of inputting the target content text information corresponding to each target vector data and the question information into the artificial intelligence model to obtain the answer result generated by the artificial intelligence model and associated with the multimedia content includes: The target content text information and the question information are input into the artificial intelligence model, and the artificial intelligence model generates the answer result associated with the multimedia content according to the corresponding content generation rule information.
5. The method according to claim 1, wherein, The step of inputting the target content text information corresponding to each target vector data and the question information into the artificial intelligence model to obtain the answer result generated by the artificial intelligence model and associated with the multimedia content includes: Obtain historical dialogue information for each round prior to the question time in the question information; the historical dialogue information is dialogue information related to the multimedia content; The historical dialogue information from each round, the target content text information, and the question information are input into the artificial intelligence model to obtain the answer results associated with the multimedia content.
6. The method according to claim 1, wherein, The multimedia content can be obtained using at least one of the following methods: The system retrieves local text or video content uploaded by the client, and obtains the text or video content from the storage location corresponding to the uploaded content link.
7. The method according to claim 2, wherein, When the multimedia content includes video content, the content text information includes image text information corresponding to each video frame image; By inputting multiple pieces of the aforementioned textual information into the artificial intelligence model, a summary report information corresponding to the multimedia content is obtained, including: The image and text information is input into the artificial intelligence model to obtain the keyframe information, keyword information and video content summary information of the video content. Based on the keyframe information, determine the keyframe images corresponding to the video content; Based on the keyframe images, the keyword information, and the video content summary information, a summary report information corresponding to the video content is generated.
8. A content generation device based on multimedia content, comprising: The parsing module is used to respond to the multimedia content uploaded by the client by calling the parsing plugin corresponding to the format of the multimedia content to parse the multimedia content and obtain multiple content text information, wherein the multimedia content includes at least one of text content in any text format and video content in any video format; The storage module is used to store the multiple content text information in the form of vector data into a vector database according to the degree of text association between the multiple content text information; the vector data in the vector database have an index relationship related to the degree of text association. A search module is configured to, in response to receiving a query related to the multimedia content, search the vector database for target vector data that match the query, based on the index relationship between vector data in the vector database; and The output module is used to input the target content text information corresponding to each target vector data and the question information into the artificial intelligence model to obtain the answer result generated by the artificial intelligence model and associated with the multimedia content. Specifically, in the step of responding to receiving a question related to the multimedia content and searching for target vector data matching the question information from the vector database based on the index relationship between the vector data in the vector database, the search module is used to: Based on the matching degree between the question information and each vector data in the vector database, find the first vector data that matches the question information; Based on the index relationship between the vector data in the vector database, find the second vector data that has an index relationship with the first vector data; The target vector data that match the question information includes the first vector data and the second vector data.
9. A computer device, comprising: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, the processor is configured to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the processor performs the steps of the content generation method based on multimedia content as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, wherein, When the computer program is run by a computer device, the computer device performs the steps of the content generation method based on multimedia content as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Information fragment associative output method and system
CN103729422A
Multimedia resource searching method and device, equipment and medium
CN115203445A
Contextual search on multimedia content
US20150370859A1
Providing a self-maintaining automated chat response generator
US20190057143A1
