Video question answering method, computer device and storage medium

Abstract

Embodiments of the present disclosure provide a video question answering method, a computer device and a storage medium. The video question answering method comprises: obtaining a question to be answered and a video to be recognized; determining, based on the question to be answered, an initial answer video segment containing caption text matched with the question to be answered and adjacent video segments to be recognized not containing caption text from the video to be recognized; predicting, based on the video segments to be recognized, the matched caption text and the initial answer video segment, whether there is semantic similarity between the content of the video segments to be recognized and the content of the initial answer video segment; and determining, according to whether there is semantic similarity between the content of the video segments to be recognized and the content of the initial answer video segment, whether the final answer video segment contains the video segments to be recognized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to, but are not limited to, the technical field of natural language processing. In particular, the present disclosure relates to a video question answering method, a computer device and a storage medium. BACKGROUND

[0002] In the current mobile Internet and big data era, video data on the network is growing explosively. As an increasingly rich information carrier medium, understanding the semantics of videos is a technology for many video intelligent applications, and has important research significance and practical application value. Video question answering (Video QA) is a task of inferring the correct answer from a video set given a question. With the progress of computer vision and natural language processing, video question answering has received more and more attention in the fields of medical health, education and knowledge, etc. SUMMARY

[0003] The following is an overview of the subject matter detailed herein. This overview is not intended to limit the scope of the claims.

[0004] In a first aspect, the embodiments of the present disclosure provide a video question answering method, comprising: obtaining a question to be answered and a video to be recognized; determining, based on the question to be answered, an initial answer video segment containing caption text matched with the question to be answered and adjacent video segments to be recognized not containing caption text from the video to be recognized; predicting, based on the video segment to be recognized, the matched caption text and the initial answer video segment, whether there is semantic similarity between the content of the video segment to be recognized and the content of the initial answer video segment; and determining whether the final answer video segment contains the video segment to be recognized according to whether there is semantic similarity between the content of the video segment to be recognized and the content of the initial answer video segment.

[0005] In an exemplary embodiment, the predicting, based on the video segment to be recognized, the matched caption text and the initial answer video segment, whether there is semantic similarity between the content of the video segment to be recognized and the content of the initial answer video segment, comprises: constructing a to-be-predicted feature vector based on the video segment to be recognized, the matched caption text and the initial answer video segment; and predicting, according to the to-be-predicted feature vector, whether there is semantic similarity between the content of the video segment to be recognized and the content of the initial answer video segment.

[0006] In an example embodiment, the constructing a to-be-predicted feature vector based on the to-be-identified video segment, the matched subtitle text, and the initial answer video segment comprises: performing feature extraction on the to-be-identified video segment to obtain a first video feature vector; performing feature extraction on the matched subtitle text to obtain a text feature vector; using an attention mechanism to perform modal fusion on the first video feature vector and the text feature vector to obtain a feature vector with integrated semantics; performing feature extraction on the initial answer video segment to obtain a second video feature vector; and splicing the second video feature vector and the feature vector with integrated semantics to obtain the to-be-predicted feature vector.

[0007] In an example embodiment, the predicting, according to the to-be-predicted feature vector, whether the content of the to-be-identified video segment and the content of the initial answer video segment have semantic similarity comprises: inputting the to-be-predicted feature vector into a first natural language model to obtain a first prediction result indicating that the content of the to-be-identified video segment and the content of the initial answer video segment have semantic similarity or a second prediction result indicating that the content of the to-be-identified video segment and the content of the initial answer video segment do not have semantic similarity.

[0008] The determining, according to whether the content of the to-be-identified video segment and the content of the initial answer video segment have semantic similarity, whether the final answer video segment contains the to-be-identified video segment comprises: when the obtained prediction result is the first prediction result, merging the initial answer video segment and the to-be-identified video segment, and determining the merged video segment as the final answer video segment; or when the obtained prediction result is the second prediction result, determining the initial answer video segment as the final answer video segment.

[0009] In an example embodiment, the obtaining a to-be-answered question and a to-be-identified video comprises: obtaining the to-be-answered question in response to a query operation; and based on the to-be-answered question, acquiring the to-be-identified video and a subtitle file corresponding to the to-be-identified video from a pre-set database.

[0010] The determining, based on the to-be-answered question, an initial answer video segment containing subtitle text matched with the to-be-answered question and a to-be-identified video segment adjacent to the initial answer video segment and not containing subtitle text from the to-be-identified video comprises: matching the to-be-answered question with the subtitle file corresponding to the to-be-identified video to obtain the matched subtitle text; and based on time information of the matched subtitle text, segmenting the initial answer video segment and the to-be-identified video segment from the to-be-identified video.

[0011] In an example embodiment, the matching the question to be answered with the subtitle file corresponding to the video to be identified, and obtaining the matched subtitle text, comprises: for each subtitle text in the subtitle file corresponding to the video to be identified, respectively performing the following operations: configuring a preset prompt text template for the question to be answered and the subtitle text, and constructing a to-be-judged text; based on the to-be-judged text, predicting whether the subtitle text is an answer matched with the question to be answered by a second natural language model; and when the subtitle text is the answer matched with the question to be answered, taking the subtitle text as the matched subtitle text.

[0012] In an example embodiment, the predicting, based on the to-be-judged text, whether the subtitle text is an answer matched with the question to be answered by a second natural language model, comprises: inputting the to-be-judged text into the second natural language model to obtain an output probability value, the probability value being used to indicate a size of semantic correlation between the question to be answered and the subtitle text; and based on whether the probability value is greater than a preset probability threshold, predicting whether the subtitle text is the answer matched with the question to be answered.

[0013] In an example embodiment, the configuring a preset prompt text template for the question to be answered and the subtitle text, and constructing a to-be-judged text, comprises: inserting the question to be answered into the preset prompt text template to obtain a to-be-spliced text; and splicing the to-be-spliced text and the subtitle text to obtain the to-be-judged text.

[0014] In an example embodiment, the obtaining, based on the question to be answered, the video to be identified and the subtitle file corresponding to the video to be identified from a pre-set database, comprises: performing semantic understanding on the question to be answered to obtain a question category of the question to be answered; taking the question category as a video category to be identified, determining, from the pre-set database, a plurality of candidate videos whose video categories are the video category to be identified and subtitle files corresponding to the plurality of candidate videos respectively; and determining, from the plurality of candidate videos and the subtitle files corresponding to the plurality of candidate videos respectively, the video to be identified and the subtitle file corresponding to the video to be identified.

[0015] In an example embodiment, the method further comprises:

[0016] obtaining a video to be extracted with subtitles;

[0017] performing frame extraction processing on the video to be extracted with subtitles to obtain a set of image frames corresponding to the video to be extracted with subtitles, the set of image frames comprising a plurality of key frames;

[0018] performing a subtitle extraction operation on each key frame in the image frame set respectively to obtain a subtitle file corresponding to the video to be extracted subtitle, wherein the subtitle extraction operation comprises: performing optical character recognition processing on the key frame to obtain coordinates of a plurality of text boxes in the key frame and a plurality of texts in the key frame; determining subtitle texts of the key frame based on the coordinates of the plurality of text boxes in the key frame and the plurality of texts in the key frame;

[0019] associating the subtitle file corresponding to the video to be extracted subtitle with the video to be extracted subtitle and storing into a database.

[0020] In an example embodiment, the determining the subtitle texts of the key frame based on the coordinates of the plurality of text boxes in the key frame and the plurality of texts in the key frame comprises: determining, from the plurality of texts in a key frame adjacent to the key frame, contrastive texts corresponding to positions of each text of the key frame; calculating a text difference degree between each text of the key frame and the corresponding contrastive text; determining a frequency of occurrence of coordinates of a text box to which each text of the key frame belongs based on the coordinates of the plurality of text boxes in the key frame; sorting the plurality of texts in the key frame based on the text difference degree between each text of the key frame and the corresponding contrastive text and the frequency of occurrence of the coordinates of the text box to which each text of the key frame belongs; and determining the subtitle texts of the key frame from the plurality of texts in the key frame according to the sorting result.

[0021] In an example embodiment, the performing the subtitle extraction operation on each key frame in the image frame set respectively to obtain the subtitle file corresponding to the video to be extracted subtitle comprises: determining text box information corresponding to each of a plurality of key frames in the image frame set by a text detection algorithm, the text box information comprising at least one of a text box number and text box coordinates; removing, based on the text box information corresponding to each of the plurality of key frames in the image frame set, key frames satisfying a preset condition from the image frame set to obtain a processed image frame set, wherein the key frames satisfying the preset condition comprise at least one of a first kind of key frame with a text box number greater than a preset number threshold, a second kind of key frame with text box coordinates outside a preset range, and a third kind of key frame without a text box; and performing the subtitle extraction operation on each key frame in the processed image frame set to obtain the subtitle file corresponding to the video to be extracted subtitle.

[0022] In an example embodiment, the method further comprises: pushing the final answer video segment, or pushing a to-be-identified video carrying identification information for indicating a position of the final answer video segment based on a start time and an end time of the final answer video segment.

[0023] In a second aspect, embodiments of this disclosure provide a computer device, including: a processor and a memory storing a computer program executable on the processor, wherein the processor is configured to implement the steps of the video question-and-answer method as described in any of the foregoing embodiments when executing the program.

[0024] Thirdly, embodiments of this disclosure provide a non-transient computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to perform the steps of the video question-and-answer method described in any of the above embodiments.

[0025] The video question-answering method, computer device, and storage medium provided in this exemplary embodiment, after obtaining the question to be answered and the video to be identified, can determine, based on the question to be answered, an initial answer video segment containing subtitle text matching the question to be answered, and its adjacent video segments to be identified that do not contain subtitle text, from the video to be identified. Then, based on the video segments to be identified, the matching subtitle text, and the initial answer video segment, it predicts whether there is semantic similarity between the content of the video segments to be identified and the content of the initial answer video segment. Finally, based on whether there is semantic similarity between the content of the video segments to be identified and the content of the initial answer video segment, it determines whether the final answer video segment contains the video segments to be identified. Thus, compared to answer video segments determined solely by subtitles, the final answer video segment determined by determining whether there is semantic similarity between the initial answer video segment containing subtitle text matching the question to be answered and its adjacent video segments to be identified that do not contain subtitle text is more complete. Therefore, the final answer video segment pushed to the client is more accurate, improving the accuracy of video answers and enhancing the interactive experience.

[0026] Other features and advantages of this disclosure will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the disclosure. Other advantages of this disclosure may be realized and obtained by means of the methods described in the description and the accompanying drawings.

[0027] After reading and understanding the accompanying diagrams and detailed descriptions, the other aspects can be understood. Attached Figure Description

[0028] The accompanying drawings are provided to illustrate the technical solutions of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure. The shapes and sizes of the components in the drawings do not reflect actual proportions and are only intended to illustrate the content of this disclosure.

[0029] Figure 1 This is a schematic diagram of a first type of video question-answering method in an exemplary embodiment of this disclosure;

[0030] Figure 2 This is a schematic diagram of a preset prompt text template in an exemplary embodiment of this disclosure;

[0031] Figure 3 This is a schematic diagram of a second flowchart of a video question-answering method in an exemplary embodiment of this disclosure;

[0032] Figure 4 This is a schematic diagram of a third process for a video question-answering method in an exemplary embodiment of this disclosure;

[0033] Figure 5 This is a schematic diagram of the fourth process of the video question-answering method in an exemplary embodiment of this disclosure;

[0034] Figure 6 This is a schematic diagram of the process for extracting video subtitles in the video question-and-answer method of the present disclosure, as an exemplary embodiment.

[0035] Figure 7 This is a flowchart illustrating an application scenario of the video question-answering method in an exemplary embodiment of this disclosure;

[0036] Figure 8 This is a schematic diagram illustrating an application scenario of the video question-answering method in an exemplary embodiment of this disclosure;

[0037] Figure 9 This is a schematic diagram of the structure of a computer device in an exemplary embodiment of this disclosure. Detailed Implementation

[0038] This document describes several embodiments, but these descriptions are exemplary and not limiting. Many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the accompanying drawings and discussed in exemplary embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or substitute for, any feature or element of any other embodiment.

[0039] In describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. Other sequences of steps are possible, as will be understood by those skilled in the art. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Furthermore, the claims relating to the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art will readily understand that these orders may be varied and still remain within the spirit and scope of the embodiments disclosed herein.

[0040] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0041] With the widespread adoption of the internet, in the current era of mobile internet and big data, online video data has experienced explosive growth, serving as an increasingly rich medium for information delivery. While plain text-based answers increase the difficulty of user comprehension, video answers, rich in visual information, are more intuitive and greatly aid in content understanding. Therefore, semantic understanding of videos is a crucial technology for many intelligent video applications, possessing significant research value and practical application significance. Video QA (Question Answering) is a task that, given a question in natural language, retrieves the correct answer from a collection of videos. With advancements in computer vision and natural language processing, video QA has a wide range of applications, such as in healthcare, education, and video retrieval, receiving increasing attention. However, given the vast amount of video resources available, enabling users to accurately find the video clips they want is paramount.

[0042] While subtitles can provide important information for video Q&A, relying solely on subtitles to determine the scope of the video's answer is inaccurate. For example, in fitness tutorial videos, the subtitles corresponding to the instructor's narration are usually synchronized with the actual movements shown in the video. However, if the subtitles appear at the beginning of the movement demonstration and then disappear while the movement continues, the video's duration will exceed the subtitles' presentation time. In such cases, the video may be prematurely cut off, resulting in incomplete video Q&A for the user, leading to inaccurate answers and a poor user experience.

[0043] In one exemplary embodiment, the terms "question to be answered" and "caption" used in this exemplary embodiment can both refer to data presented in text form. The term "text" can represent characters of any language and form, such as Chinese, English, or other language characters, numbers, and combinations thereof. The term "text box" can refer to the image area containing the text.

[0044] In one exemplary embodiment, the terms "image frame" and "keyframe" used in the exemplary embodiments of this disclosure can both refer to data presented in image form.

[0045] In one exemplary embodiment, the terms "video to be identified," "candidate video," "initial answer video segment," "video segment to be identified," "final answer video segment," and "video from which subtitles are to be extracted," used in the exemplary embodiments of this disclosure, can all refer to data presented in video form.

[0046] This disclosure provides a video question-answering method that determines whether the final answer video segment contains the video segment to be identified by predicting whether there is semantic similarity between the content of an initial answer video segment containing subtitle text matching the question to be answered and the content of an adjacent video segment to be identified that does not contain subtitle text. Therefore, compared to some technologies that only use subtitles to determine the range of video answers, this method avoids the problem of inaccurate video question-answering localization and segmentation, thereby improving the accuracy of video answers and enhancing the interactive experience.

[0047] In one exemplary embodiment, the video question-answering method can be applied to a video question-answering device. For example, when deployed on a terminal, server, or other processing device, the video question-answering device can perform functions such as video querying. Exemplarily, the terminal can be a user equipment (UE), mobile device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device, etc. In some possible implementations, the video question-answering method can also be implemented by a processor calling a computer program stored in memory.

[0048] In one exemplary embodiment, the video question-answering method can be applied to many application areas such as healthcare, educational knowledge, or smart education.

[0049] Figure 1 This is a schematic diagram of a first type of video question-answering method in an exemplary embodiment of this disclosure, as shown below. Figure 1 As shown, this video question-and-answer method may include the following steps:

[0050] Step 11: Obtain the questions to be answered and the video to be identified;

[0051] Step 12: Based on the question to be answered, identify the initial answer video segment containing caption text that matches the question to be answered, and its adjacent video segments to be identified that do not contain caption text.

[0052] Step 13: Based on the video segment to be identified, the matched subtitle text, and the initial response video segment, predict whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment;

[0053] Step 14: Determine whether the final response video segment contains the video segment to be identified based on whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment.

[0054] In one exemplary embodiment, the video segment to be identified may include at least one image frame that does not contain subtitle text. For example, in chronological order, the video segment to be identified may include at least one image frame that does not contain subtitle text, and this image frame may be located before the first image frame of the initial response video segment or after the last image frame of the initial response video segment, etc. For example, the number of video segments to be identified may be one or more.

[0055] In one exemplary embodiment, the number of initial response video segments can be one or more.

[0056] In one exemplary embodiment, the number of caption texts in the initial answer video segment that match the question to be answered can be one or more.

[0057] In one exemplary embodiment, the caption text matching the question to be answered can correspond to at least one image frame. For example, the caption text contained in a 2-second video clip can be the same, in which case one caption text can correspond to multiple image frames in the video clip.

[0058] In one exemplary embodiment, when the content of the video segment to be identified is semantically similar to the content of the initial response video segment, it can be determined that the final response video segment includes the video segment to be identified. In this case, the video segment to be identified and the initial response video segment can be used as the final response video segment. Alternatively, when the content of the video segment to be identified is not semantically similar to the content of the initial response video segment, it can be determined that the final response video segment does not include the video segment to be identified. In this case, the initial response video segment can be used as the final response video segment. Thus, the obtained final response video segment is more accurate.

[0059] The video question-answering method provided in this exemplary embodiment, after obtaining the question to be answered and the video to be identified, can determine, based on the question to be answered, an initial answer video segment containing subtitle text matching the question to be answered, and its adjacent video segments to be identified that do not contain subtitle text, from the video to be identified. Then, based on the video segments to be identified, the matching subtitle text, and the initial answer video segment, it predicts whether there is semantic similarity between the content of the video segments to be identified and the content of the initial answer video segment. Finally, based on whether there is semantic similarity between the content of the video segments to be identified and the content of the initial answer video segment, it determines whether the final answer video segment contains the video segments to be identified. Thus, since the question to be answered is presented directly in text form, the subtitle text of the video to be identified can be directly used to perform semantic retrieval and matching to determine the initial answer video segment containing subtitle text matching the question to be answered. For video segments to be identified that do not contain subtitles, they cannot be directly retrieved using the question to be answered because there are no directly corresponding subtitles. However, the inventors of this disclosure have discovered that the content of video segments without subtitles has a certain semantic correlation with the content of adjacent video segments with subtitles. Thus, by checking whether there is semantic similarity between the initial answer video segment containing subtitles that match the question to be answered and its adjacent video segments to be identified that do not contain subtitles, the final answer video segment determined is more complete. This makes the matching degree between the final answer video segment and the question to be answered higher, which can avoid the problem of inaccurate video question-and-answer positioning and segmentation. Therefore, it can improve the accuracy of video answers, improve the interaction effect, and improve the user experience.

[0060] In one exemplary embodiment, step 11 may include:

[0061] Step 111: In response to the query operation, obtain the questions to be answered;

[0062] Step 112: Based on the questions to be answered, retrieve the video to be identified and the corresponding subtitle file from the pre-set database.

[0063] In one exemplary embodiment, the subtitle file may include at least one subtitle text and at least one timestamp of the subtitle text, wherein the timestamp of the subtitle text may refer to the playback time of the subtitle text in the video, including the start time and end time. For example, the subtitle file corresponding to a certain video may be ["untext":1s-11s], ["text1":10s-12s], ["untext":12s-14s], ["text2":14s-17s], indicating that there is no subtitle text from the 1st to the 11th second, the subtitle text from the 10th to the 12th second is text1, there is no subtitle text from the 12th to the 14th second, and the subtitle text from the 14th to the 17th second is text2.

[0064] For example, when a user needs to perform a video search, they can do so through a client. The server then receives the question input by the user. Next, the server can retrieve the video to be identified and its corresponding subtitle file from a pre-set database based on the question.

[0065] Exemplary, in the exemplary embodiments of this disclosure, a server is used as the execution entity. The server receives user input of questions to be answered. For example, the server may receive user input of questions to be answered in voice form, and then convert the voice-to-text format of the questions to be answered; or, the server may directly receive user input of questions to be answered in text form.

[0066] Here, a pre-set database can store multiple videos, and subtitles are extracted from multiple videos in advance to obtain subtitle files corresponding to each video. The multiple videos and their corresponding subtitle files are then stored in the pre-set database.

[0067] In one exemplary embodiment, step 112 may include:

[0068] Step 1121: Perform semantic understanding on the question to be answered to obtain the question category;

[0069] Step 1122: Using the problem category as the video category to be identified, determine multiple candidate videos whose video category is the video category to be identified and their corresponding subtitle files from a pre-set database;

[0070] Step 1123: From multiple candidate videos and their respective subtitle files, determine the video to be identified and its corresponding subtitle file.

[0071] Thus, by using global semantic understanding to determine the video domain (i.e., video category) involved in the question to be answered, the search scope can be narrowed, thereby speeding up the search process.

[0072] In one exemplary embodiment, step 1121 may include: performing semantic understanding on the question to be answered using a conventional classification algorithm or a pre-trained language model to determine the question category. Of course, in addition to the two methods listed above, other algorithms or models capable of solving binary classification problems can also be used when performing semantic understanding on the question to be answered. This disclosure does not limit this approach.

[0073] For example, a user asks a question—"Instructional videos for back exercises"—and through global semantic understanding, it can be determined that the question is most likely in the fitness category. Therefore, the question category can be identified as "fitness." Consequently, the videos the user wants to search for are related to the fitness category. In subsequent video searches, the user can then locate the relevant video among multiple candidate videos in the fitness category. This narrows the search scope and thus speeds up the search process.

[0074] In one exemplary embodiment, step 1123 may include: for each candidate video, based on the question to be answered and the subtitle files corresponding to each of the multiple candidate videos, obtaining the matching subtitle text corresponding to each of the multiple candidate videos by performing subsequent step 121; and determining the candidate videos in the multiple candidate videos whose number of matching subtitle texts is greater than a preset threshold as the videos to be identified.

[0075] In another exemplary embodiment, step 1123 may include: for each candidate video, based on the question to be answered and the subtitle files corresponding to each of the candidate videos, obtaining the matching subtitle text corresponding to each of the candidate videos by performing subsequent step 121; sorting the candidate videos in descending order according to the number of matching subtitle texts corresponding to each candidate video; and determining the candidate video with the highest ranking as the video to be identified. Thus, by selecting candidate videos with higher relevance (such as candidate videos with more than a preset threshold of matching subtitle texts, or candidate videos with the largest number of matching subtitle texts) as the videos to be identified, the retrieval speed of video question answering can be improved.

[0076] In one exemplary embodiment, step 12 may include:

[0077] Step 121: Match the questions to be answered with the subtitle file to obtain the subtitle text that matches the questions to be answered;

[0078] Step 122: Based on the caption text matching the question to be answered, segment the initial answer video segment and the video segment to be identified from the video to be identified.

[0079] In one exemplary embodiment, the number of subtitle texts in the subtitle file of the video to be identified can be one or more. The number of subtitle texts in the subtitle file of the video to be identified that match the question to be answered can also be one or more.

[0080] In one exemplary embodiment, step 121 may include:

[0081] Step 1211: For each subtitle text in the subtitle file, perform the following operations:

[0082] Step 1212: Configure preset prompt text templates for the question to be answered and the subtitle text to construct the text to be judged;

[0083] Step 1213: Based on the text to be judged, predict whether the subtitle text is the answer to the question to be answered using a second natural language model;

[0084] For example, a second natural language model can be pre-trained based on question samples configured with prompt text templates and their corresponding answer samples.

[0085] Step 1214: When the caption text is the answer to the question to be answered, use the caption text as the matching caption text.

[0086] In this way, the subtitle text that matches the question to be answered can be determined from the subtitle file of the video to be identified.

[0087] In one exemplary embodiment, step 1212 may include: inserting the question to be answered into a preset prompt text template to obtain text to be concatenated; concatenating the text to be concatenated with the subtitle text to obtain text to be judged.

[0088] Here, the preset prompt text template can refer to the Prompt paradigm in the field of natural language processing. The Prompt paradigm can fully utilize the prior knowledge of the natural language model, so that the upstream task of the natural language model matches the downstream learning task, and its training results are significantly higher than those of the transfer learning mode.

[0089] For example, such as Figure 2 As shown, the preset prompt text template can be designed as "The Answer of question is Subtitle", where "question" is the position where the question to be answered is inserted, and "Subtitle" is the position where each subtitle text in the subtitle file corresponding to the video to be recognized is spliced ​​together.

[0090] In one exemplary embodiment, the text length Max_length of the text to be judged is 512. Of course, the text length of the text to be judged can also be set to other values, and this embodiment of the present disclosure does not limit this.

[0091] In one exemplary embodiment, step 1213 may include: inputting the text to be judged into a second natural language model to obtain an output probability value, the probability value being used to indicate the magnitude of the semantic relevance between the question to be answered and the subtitle text; and predicting whether the subtitle text is an answer matching the question to be answered based on whether the probability value is greater than a preset probability threshold.

[0092] For example, the preset probability threshold can be set to 0.5. After inputting the text to be judged into the second natural language model, if the probability value output by the second natural language model is greater than 0.5, then the subtitle text can be determined to be one of the answers to the question to be answered. Alternatively, if the probability value output by the second natural language model is not greater than 0.5, then the subtitle text can be determined not to be an answer to the question to be answered.

[0093] Thus, after configuring the text of the question to be answered and each subtitle text in the subtitle file of the video to be identified with a preset prompt text template, each corresponding text to be judged is obtained. Each of these texts is then fed into a pre-trained second natural language model to predict whether each subtitle text in the subtitle file of the video to be identified is the answer to the question. This can be transformed into a typical binary classification problem. Next, the highly relevant and densely packed subtitle regions in all the subtitles of the video to be identified can be identified as the subtitle text in the video to be identified that matches the question. Therefore, the coarse-grained range of the video question-and-answer process can be determined, i.e., the initial answer video segment can be obtained.

[0094] Here, directly determining the initial answer video segment from the video to be identified based on the timestamp of the subtitle text that matches the question to be identified as the time range corresponding to the final video Q&A may result in some videos being segmented too early or the content displayed in the video being directly truncated, which may easily lead to inaccurate video Q&A positioning and segmentation. Therefore, in order to improve the accuracy of video answers, after determining the initial answer video segment that matches the subtitle text that matches the question to be answered, steps 13 to 14 can be performed to determine the accurate final answer video segment.

[0095] In one exemplary embodiment, step 13 may include:

[0096] Step 131: Construct the feature vector to be predicted based on the video segment to be identified, the matched subtitle text, and the initial response video segment;

[0097] Step 132: Based on the feature vector to be predicted, predict whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment.

[0098] In one exemplary embodiment, step 131 may include:

[0099] Step 1311: Extract features from the video segment to be identified to obtain the first video feature vector;

[0100] Step 1312: Extract features from the matched subtitle text to obtain text feature vectors;

[0101] Step 1313: Using an attention mechanism, modal fusion is performed on the first video feature vector and the text feature vector to obtain a feature vector incorporating semantics;

[0102] Step 1314: Extract features from the initial answer video segment to obtain the second video feature vector;

[0103] Step 1315: Concatenate the second video feature vector and the semantically integrated feature vector to obtain the feature vector to be predicted.

[0104] In some exemplary embodiments, before modal fusion of the video feature vector and the text feature vector, the video response method may further include: processing the video feature vector and / or the text feature vector so that the dimensions of the video feature vector and the text feature vector are the same when modally fusing the video feature vector and the text feature vector. For example, step 1312 may include: step 1312a: vectorizing the matched subtitle text to obtain an initial text feature vector Text_Feature; step 1312b: using a linear layer (Linear Year) to linearly transform the initial text feature vector Text_Feature to obtain a text feature vector K. Thus, by mapping the initial text feature vector to the same dimension as the first video feature vector, the text feature vector K is obtained, which facilitates subsequent modal fusion of the two modalities.

[0105] In some exemplary embodiments, before concatenating the video feature vector with the text feature vector, the video response method may further include processing the video feature vector and / or the text feature vector so that when concatenating the video feature vector with the text feature vector, the dimensions of the video feature vector and the text feature vector are the same.

[0106] In one exemplary embodiment, such as Figure 3 As shown, step 131 may include:

[0107] Step 1311: Extract features from the video segment to be identified to obtain the first video feature vector V;

[0108] Step 1312a: Vectorize the matched subtitle text to obtain the initial text feature vector Text_Feature;

[0109] Step 1312b: Use a linear layer to perform a linear transformation on the initial text feature vector Text_Feature to obtain the text feature vector K;

[0110] Thus, by mapping the initial text feature vector Text_Feature to the same dimension as the first video feature vector V, we obtain the text feature vector K, which facilitates the subsequent line modality fusion of information from the two modalities.

[0111] Step 1313: Using an attention mechanism, modally fuse the first video feature vector V and the text feature vector K to obtain a feature vector incorporating semantics.

[0112] Step 1314: Extract features from the initial answer video segment to obtain the second video feature vector J;

[0113] Step 1315: Combine the second video feature vector J with the semantically integrated feature vector. The features are concatenated to obtain the feature vector to be predicted.

[0114] In one exemplary embodiment, to obtain video feature vectors, video feature extraction can employ a ResNeXt model, or a ResNet model, etc. The ResNeXt model is a neural network, its predecessor being the Residual Network (ResNet) model.

[0115] In one exemplary embodiment, such as Figure 3 As shown, taking the ResNeXt model for video feature extraction as an example, step 1311 may include: using the ResNeXt model to extract features from the video segment to be identified, and obtaining the first video feature vector V.

[0116] In this context, "modality" refers to the source or form of information. For example, video feature vectors are one type of modality data, while text feature vectors are another. Modality fusion (also known as intermodal attention processing) refers to applying attention to at least two types of data to facilitate information interaction between different modalities, thereby highlighting the correlation between the two modalities.

[0117] In one exemplary embodiment, modality fusion can be an attention processing method based on commonly used attention mechanisms, such as cross-attention networks, key-value pair attention mechanisms, or multi-head attention mechanisms. Alternatively, modality fusion can be implemented based on custom formulas or algorithms.

[0118] In one exemplary embodiment, such as Figure 3 As shown, step 1314 may include: using a cross-attention mechanism to modally fuse the first video feature vector V and the text feature vector K to obtain a semantically integrated feature vector. Here, the semantically integrated feature vector is a multimodal feature vector.

[0119] The cross-attention mechanism refers to a deep neural network that incorporates an attention mechanism to achieve explicit feature combination of a specified order with linear space complexity. In this embodiment, the cross-attention mechanism can perform modal fusion of the first video feature vector and the text feature vector based on the correlation between the first video feature vector and the text feature vector to obtain a feature vector incorporating semantics. For example, in the cross-attention mechanism, both the query vector and the key vector are represented by the text feature vector K, and the value vector is represented by the first video feature vector V.

[0120] Thus, since the matched subtitle text contains semantic information that is helpful for the video response task, the feature vectors generated by modal fusion that incorporate semantics (i.e., multimodal feature vectors) can play a good auxiliary role in the subsequent segmentation of video response segments, thereby improving the accuracy of video responses.

[0121] In an exemplary embodiment, taking the ResNeXt model for video feature extraction as an example, step 1315 may include: extracting features from the initial response video segment using the ResNeXt model to obtain a second video feature vector.

[0122] In one exemplary embodiment, step 1316 may include: using the concat function to concatenate the second video feature vector and the semantically integrated feature vector to obtain the feature vector to be predicted.

[0123] In one exemplary embodiment, step 132 may include: inputting the feature vector to be predicted into a first natural language model to obtain a first prediction result indicating that the content of the video segment to be identified has semantic similarity with the content of the initial response video segment, or a second prediction result indicating that the content of the video segment to be identified does not have semantic similarity with the content of the initial response video segment.

[0124] In one exemplary embodiment, the first natural language model may employ a Recurrent Neural Network (RNN) combined with a Conditional Random Field (CRF). For example, taking an RNN employing a Long Short-Term Memory (LSTM) network, then, as... Figure 3 As shown, the first natural language model can be implemented using LSTM+CRF, where the feature vector to be predicted is used as the input of LSTM, the output of LSTM can be used as the input of CRF, and the output of CRF is used as the overall prediction result of the first natural language model.

[0125] In one exemplary embodiment, step 14 may include: when the obtained prediction result is a first prediction result, indicating that the final answer video segment contains the video segment to be identified, that is, the video segment to be identified is part of the answer required by the user, in this case, merging the initial answer video segment and the video segment to be identified, and determining the merged video segment as the final answer video segment; or, when the obtained prediction result is a second prediction result, indicating that the final answer video segment does not contain the video segment to be identified, that is, the video segment to be identified is not part of the answer required by the user, in this case, directly determining the initial answer video segment as the final answer video segment. Thus, the determined final answer video segment is more accurate.

[0126] Figure 4 This is a schematic diagram of a third process for a video question-answering method in an exemplary embodiment of this disclosure. For example... Figure 4 As shown, after step 14, the video response method may further include:

[0127] Step 41: Push the final answer video clip, or, based on the start and end times of the final answer video clip, push the video to be identified carrying identification information indicating the location of the final answer video clip.

[0128] For example, based on the start and end times of the final response video segment, the BIO annotation framework is used to annotate the video to be identified. The annotation information can be B-part, I-part, or Other, where B-part represents the start frame of the final response video segment, I-part represents the last frame of the final response video segment, and Other represents non-final response video segments.

[0129] Thus, by directly pushing the final answer video clip itself, the client can quickly display the final answer video clip without user intervention. Alternatively, by pushing a video to be identified that carries identification information indicating the location of the final answer video clip, the user can quickly locate the final answer video clip by interacting with the identification information.

[0130] Subtitles in a video provide explanations and descriptions of the video content, essentially covering most of what's shown, making them a crucial source of information. Currently, two main methods are used for extracting video subtitles: one is extraction based on the specific location of a given subtitle frame. While this eliminates interference from non-subtitle text, it requires manual positioning for each video when subtitle positions are not fixed. The second method extracts subtitles by generating text from the video's audio. However, this method is prone to accuracy issues when the pronunciation of characters in the audio is inaccurate, still requiring manual correction. Clearly, for extracting subtitles from large batches of personalized videos, these two methods significantly increase workload, are time-consuming and labor-intensive, have low efficiency, and high labor costs. Therefore, neither method is suitable for extracting subtitles from large volumes of videos.

[0131] Figure 5 This is a schematic diagram of the fourth process of the video question-answering method in an exemplary embodiment of this disclosure. For example... Figure 5 As shown, prior to step 11, the video question-and-answer method may further include:

[0132] Step 51: Obtain the video from which subtitles are to be extracted;

[0133] Step 52: Perform frame extraction on the video from which subtitles are to be extracted to obtain the image frame set corresponding to the video from which subtitles are to be extracted. The image frame set may include multiple keyframes.

[0134] Step 53: Perform subtitle extraction on each keyframe in the image frame set to obtain the subtitle file of the video from which subtitles are to be extracted; wherein, the subtitle extraction operation may include the following steps 531 to 532:

[0135] Step 531: Perform Optical Character Recognition (OCR) processing on the keyframe to obtain the coordinates of multiple text boxes and multiple texts in the keyframe;

[0136] Step 532: Determine the caption text of the keyframe based on the coordinates of multiple text boxes and multiple texts in the keyframe.

[0137] Step 54: Associate the subtitle file corresponding to the video from which subtitles are to be extracted with the video from which subtitles are to be extracted, and store them in the database.

[0138] Therefore, after obtaining the video from which subtitles are to be extracted, frame extraction can be performed on the video to obtain a set of image frames corresponding to the video from which subtitles are to be extracted. The set of image frames includes multiple keyframes. Then, after performing the above subtitle extraction operation on multiple keyframes in the set of image frames, the subtitle files corresponding to each of the multiple keyframes can be extracted, thus obtaining the subtitle files corresponding to the video from which subtitles are to be extracted. In this way, before video retrieval, the subtitles of all videos from which subtitles are to be extracted can be extracted, and the videos and their corresponding subtitle files can be put into a database for unified storage, so as to be used for video retrieval in subsequent video question-and-answer sessions.

[0139] In one exemplary embodiment, the keyframe can be any image frame in the video. For example, it can be an image frame containing text, or an image frame not containing text, or an image frame containing subtitle text, or an image frame not containing subtitle text, etc. Here, the embodiments of this disclosure do not limit it in this way.

[0140] In one exemplary embodiment, the VideoCapture function in the OpenCV2 library can be used to perform frame extraction on the video from which subtitles are to be extracted.

[0141] In one exemplary embodiment, to improve the speed of keyframe extraction, keyframes can be acquired at set intervals. For example, 3 frames can be extracted per second as keyframes, or 1 frame can be extracted every 8 seconds. For example, taking the extraction of one frame every 8 frames as an example, the frame rate represents how many frames of images are in one second. Thus, according to the formula Frame count = Frame rate / 8, the total number of frames that can be extracted from each second of video can be calculated.

[0142] In one exemplary embodiment, optical character recognition (OCR) processing may include a text detection algorithm and a text recognition algorithm.

[0143] In one exemplary embodiment, step 531 may include:

[0144] Step 5311: Use a text detection algorithm to perform text detection on the keyframe and determine the coordinates of multiple text boxes in the keyframe;

[0145] Step 5312: Use a text recognition algorithm to perform text recognition on multiple text boxes in the keyframe to obtain multiple texts in the keyframe.

[0146] In an exemplary embodiment, the text detection algorithm may adopt an algorithm such as Differentiable Binarization (DB). Exemplarily, step 5311 may include: performing text detection on the key frame through the DB algorithm to determine the coordinates (X_min, X_max, Y_min, Y_max) of the text box, where X_min represents the minimum abscissa of the text box in the current key frame, X_max represents the maximum abscissa of the text box in the current key frame, Y_min represents the minimum ordinate of the text box in the current key frame, and Y_max represents the maximum ordinate of the text box in the current key frame. For example, the parameter det_db_unclip_ratio = 1.6 in the DB algorithm.

[0147] In an exemplary embodiment, the text recognition algorithm may adopt an algorithm such as Region Convolutional Neural Network (RCNN). Exemplarily, step 5312 may include: using the RCNN algorithm to recognize the text in multiple text boxes in the key frame to obtain multiple texts in the key frame.

[0148] In an exemplary embodiment, both the text detection algorithm (such as the DB algorithm) and the text recognition algorithm (such as the RCNN deep learning model algorithm) can be implemented through the paddle framework. Among them, the paddle framework is an open-source deep learning framework that integrates the core training and inference framework of deep learning, the basic model library, the end-to-end development kit, and rich tool components.

[0149] In an exemplary embodiment, since the position of the text box (such as the subtitle box) may fluctuate within a small range after frame extraction, therefore, the text boxes in all key frames in the image frame set can be uniformly replaced with the same coordinates within a certain difference range according to ‖Y_(min,1)-Y_(min,1)‖ < a for subsequent text deduplication. Among them, ||*|| represents the norm, Y_(min,1)-Y_(min,1) represents the difference between the minimum ordinates of the text box in two adjacent key frames, and a represents the preset difference.

[0150] In an exemplary embodiment, before step 532, the method further includes: counting the frequency of occurrence of the coordinates of the text boxes in all key frames in the image frame set corresponding to the video for which subtitles are to be extracted. Correspondingly, step 532 may include:

[0151] Step 5321: Determine the comparison text corresponding to the position of each text in the key frame from the multiple texts in the key frame adjacent to the key frame;

[0152] Step 5322: Calculate the text difference between each text in the keyframe and its corresponding contrast text;

[0153] Step 5323: Based on the coordinates of multiple text boxes in the keyframe, determine the frequency of the coordinates of the text box to which each text in the keyframe belongs.

[0154] Step 5324: Sort the multiple texts in the keyframe based on the text difference between each text in the keyframe and its corresponding comparison text, as well as the frequency of the coordinates of the text box to which each text in the keyframe belongs.

[0155] Step 5325: Based on the sorting results, determine the subtitle text of the keyframe from the multiple texts in the keyframe.

[0156] In one exemplary embodiment, a keyframe adjacent to the keyframe can refer to either the preceding keyframe or the following keyframe. For example, step 5321 may include determining, from multiple texts in the preceding or following keyframe, contrast text corresponding to the position of each text in the keyframe.

[0157] In one exemplary embodiment, the text difference (Text_num, also known as the number of text differences) between two texts within the same position range of two consecutive keyframes can be measured using edit distance, which represents the minimum number of single-character edits (insertion, deletion, or replacement) required to transform one sequence of text characters into another. For example, the text difference between two texts within the same position range of two consecutive keyframes can refer to the edit distance between a text in the current keyframe and another text within the same position range of the previous or next keyframe.

[0158] In one exemplary embodiment, the text difference (Text_num) and text box frequency (Text_box) for each text in the keyframe are obtained. num Then, step 5334 may include: for each keyframe, first sorting the text boxes according to their frequency (Text_box). num Sort the text, then sort it according to the frequency of each text box (Text_box). numThe text is then sorted. Next, step 5335 may include: extracting the first sorted text from the sorting results as the current text; determining whether the text difference index (Text_num) of the remaining texts is less than half the text difference index of the current text; when the text difference index (Text_num) of the remaining texts is less than half the text difference index of the first text, extracting the second sorted text from the sorting results as the current text, and re-determining whether the text difference index (Text_num) of the remaining texts is less than half the text difference index of the current text, until the text difference index (Text_num) of the remaining texts is greater than half the text difference index (Text_num) of the current text, then using this current text as the subtitle text for that keyframe. This avoids interference from the real-time changing program markers of some video segments on the statistical analysis of the text difference index of two texts within the same position range of consecutive keyframes, thus improving the accuracy of subtitle extraction.

[0159] In one exemplary embodiment, step 53 may further include:

[0160] Step 534: Using a text detection algorithm, determine the text box information corresponding to each of the multiple keyframes in the image frame set. The text box information includes at least one of the following: the number of text boxes and the coordinates of the text boxes.

[0161] Step 535: Based on the text box information corresponding to each of the multiple keyframes in the image frame set, remove the keyframes that meet the preset conditions from the image frame set to obtain the processed image frame set; wherein, the keyframes that meet the preset conditions include at least one of the following: a first type of keyframe with a text box number greater than a preset threshold, a second type of keyframe with text box coordinates outside a preset range, and a third type of keyframe without a text box.

[0162] Step 536: Perform subtitle extraction operation on each keyframe in the processed image frame set to obtain the subtitle file corresponding to the video from which the subtitles are to be extracted.

[0163] In one exemplary embodiment, if the number of text boxes T_box_num in the first keyframe is greater than a preset threshold, it indicates that the first keyframe is formed by official documents or background text and has no reference value. In this case, the first keyframe can be filtered out, thereby improving the subtitle extraction speed. For example, the preset threshold can be set to 15. Of course, the preset threshold can be set to other values ​​by those skilled in the art according to the actual situation. Here, this embodiment does not limit this.

[0164] In one exemplary embodiment, the position of the subtitle within an image frame is generally within a certain preset range, while the coordinates of the text boxes in the second type of keyframe are all outside the preset range. This indicates that the text in the second type of keyframe is non-subtitle text and has no reference value. In this case, the second type of keyframe can be filtered out, thereby improving the subtitle extraction speed. For example, the preset range can be set to the lower half of the image frame. Of course, the preset range can be set to other ranges by those skilled in the art according to the actual situation. Here, the embodiments of this disclosure do not limit this.

[0165] In one exemplary embodiment, since the third keyframe does not have a text box, it indicates that there is no text in the third keyframe. In this case, the third keyframe can be filtered out without text recognition, thus improving the subtitle extraction speed.

[0166] Therefore, before extracting subtitles, multiple keyframes in the image frame set corresponding to the video from which subtitles are to be extracted can be filtered out to remove redundant image frames, which can improve the subtitle extraction speed.

[0167] Figure 6 This is a schematic diagram illustrating the process of extracting video subtitles in a video question-and-answer method according to an exemplary embodiment of this disclosure. Figure 6 As shown, the process of extracting video subtitles in the video question-answering method of this exemplary embodiment may include:

[0168] Step S601: Obtain the video from which subtitles are to be extracted;

[0169] Step S602: Extract keyframes from the video from which subtitles are to be extracted, and obtain multiple keyframes corresponding to the video;

[0170] Step S603: Using OCR technology, perform text recognition on multiple keyframes to determine whether there are no text boxes in the multiple keyframes;

[0171] Step S604: When there is no text box in a keyframe, it indicates that no subtitle text was recognized for that keyframe;

[0172] Step S605: When a text box exists in a keyframe, count the number of text boxes T_box_num corresponding to each of the multiple keyframes;

[0173] Step S606: Filter out keyframes with a text box count T_box_num greater than 15 from multiple keyframes;

[0174] Step S607: For the remaining key frames, according to ‖Y_(min,1)-Y_(min,1)‖<a, within a certain difference range, uniformly replace the vertical coordinates Y_axis of the text boxes corresponding to multiple key frames with the same coordinate;

[0175] Step S608: Remove duplicate texts for the texts corresponding to multiple key frames;

[0176] Step S609: Use the edit distance to measure the number of text differences between any two texts that belong to adjacent key frames and have the same position among multiple key frames, and obtain the text difference degree Text_num corresponding to each text in multiple key frames;

[0177] Step S610: Based on the coordinates of each text box in multiple key frames, determine the occurrence frequency of the coordinates of the text boxes with the same position, and obtain the text box frequency Text_box corresponding to the text box to which each text in multiple key frames belongs num ;

[0178] Step S611: For each key frame, first sort according to the text box frequency Text_box of each text num and then sort according to the text box frequency Text_box of each text num From the sorting result, take the first text Max{Text_num}(Max{Text_box num}) as the current text, and determine whether the text difference degree Text_num of the remaining texts is less than 1 / 2 of the text difference number of the current text; when the text difference degree Text_num of the remaining texts is greater than 1 / 2 of the text difference number of the first text, take the first sorted text Max{Text_num}(Max{Text_box num}) as the subtitle text of this key frame. Or, when the text difference degree Text_num of the remaining texts is less than 1 / 2 of the text difference number of the first text, from the sorting result, take the second sorted text as the current text, and re-determine whether the text difference degree Text_num of the remaining texts is less than 1 / 2 of the text difference number of the current text, until the text difference degree Text_num of the remaining texts is greater than 1 / 2 of the text difference degree of the current text, and take the current text as the subtitle text of this key frame.

[0179] Keep looping until obtaining the subtitle texts corresponding to multiple key frames respectively, and output the subtitle file corresponding to the video to be extracted with subtitles.

[0180] Thus, this video subtitle extraction process can achieve both real-time processing of small batches of videos and offline processing of large batches of videos.

[0181] With the continuous development of new media technologies, online classes have gradually emerged, and the internet is flooded with a wealth of teaching resources. While abundant teaching videos can help students consolidate their knowledge offline, a lack of focused review can actually increase time costs. When students review parts they didn't understand, they need to rewatch the videos or manually locate the relevant information, both of which waste time. For students unfamiliar with the interconnectedness of knowledge, they may even struggle to locate specific segments of professional knowledge, undoubtedly adding even more time.

[0182] The video question-and-answer method provided in this disclosure can be applied to the field of smart education, for example, in online classroom systems. As an auxiliary teaching tool, it can help users (such as students and parents) quickly search for knowledge explanation videos and accurately locate the explanation of knowledge points, effectively improving learning efficiency. Alternatively, it can help users (such as preschool children and parents) quickly search for story videos and accurately locate interesting video clips.

[0183] The following description uses the video question-and-answer method provided in this disclosure as an example of its application in the field of smart education to illustrate the video question-and-answer method provided in this disclosure. Figure 7 This is a schematic diagram illustrating the application of the video question-answering method in an exemplary embodiment of this disclosure. For example... Figure 7 As shown, the video question-and-answer method provided in this embodiment may include:

[0184] Step S701: When a student wants to watch a video about a specific knowledge point, such as an explanation of the formula method for linear equations in two variables, the question to be answered will be "Explanation of the formula method for linear equations in two variables". You can directly proceed to step S704, or first execute steps S702 to S703 and then execute step S704.

[0185] Step S702: Perform semantic understanding on the question to be answered, "Explanation of the formula method in a linear equation in two variables", and determine that the question category is mathematics.

[0186] Step S703: Using mathematics as the video category to be identified, search for videos in the pre-stored mathematics category in a pre-set database to obtain multiple candidate videos and their corresponding subtitle files. In this way, by determining the video category, the search scope can be narrowed, thus speeding up the retrieval process to some extent.

[0187] Step S704: Based on the question to be answered, multiple candidate videos, and the corresponding subtitle files for each candidate video, perform the following video question-and-answer retrieval operation:

[0188] Based on the obtained unanswered question "Explanation of the formula method for a linear equation in two variables" and the subtitle files corresponding to multiple candidate videos, the matching subtitle text corresponding to each of the multiple candidate videos is obtained.

[0189] Based on the number of matching subtitle texts corresponding to each of the multiple candidate videos, the video to be identified and the subtitle file corresponding to the video to be identified are determined from the multiple candidate videos and the subtitle files corresponding to each of the multiple candidate videos.

[0190] For each subtitle text in the subtitle file corresponding to the video to be identified, the following operations are performed: A preset prompt text template is configured for the question to be answered and the subtitle text to construct the text to be judged; based on the text to be judged, a second natural language model is used to predict whether the subtitle text is a matching subtitle text. The second natural language model is pre-trained based on question samples with configured prompt text templates and their corresponding answer samples; when the subtitle text is an answer matching the question to be answered, the subtitle text is considered a matching subtitle text.

[0191] From the video to be identified, determine the initial answer video segment containing caption text that matches the question to be answered, and its adjacent video segments to be identified that do not contain caption text.

[0192] Based on the video segment to be identified, the matched subtitle text, and the initial response video segment, predict whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment.

[0193] Based on whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment, it is determined whether to use the video segment to be identified and the initial response video segment as the final response video segment, or to use the initial response video segment as the final response video segment.

[0194] Step S705: As Figure 8 As shown, based on the start and end times of the final answer video segment, a video to be identified is pushed, carrying identification information indicating the location of the final answer video segment. In this way, users can directly click to play the video segment corresponding to "Explanation of the Formula Method in Two Linear Equations".

[0195] The descriptions of the above application scenario embodiments are similar to those of the above video question-answering method embodiments, and have similar beneficial effects. For technical details not disclosed in the application scenario embodiments of this disclosure, those skilled in the art should refer to the descriptions in the method embodiments of this disclosure for understanding, and will not be repeated here.

[0196] Based on the same inventive concept, this disclosure also provides a computer device, including: a processor and a memory storing a computer program executable on the processor, wherein the processor is configured to implement the steps of the video question-and-answer method in one or more of the above exemplary embodiments when executing the program.

[0197] In one exemplary embodiment, such as Figure 9 As shown, the computer device 90 may include: at least one processor 91; and at least one memory 92 and bus 93 connected to the processor 91; wherein the processor 91 and the memory 92 communicate with each other through the bus 93; the processor 91 is configured to call program instructions in the memory 92 to execute the steps of the video question-and-answer method in one or more exemplary embodiments described above.

[0198] In one exemplary embodiment, the processor may be, for example, a Central Processing Unit (CPU), other general-purpose processors, a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, application-specific integrated circuits (ASICs), or other chips with computing functions. The general-purpose processor may be a Microprocessor Unit (MPU), or it may be any conventional processor. This disclosure does not limit the scope of the embodiment.

[0199] In one exemplary embodiment, the memory may include non-permanent memory and / or non-volatile memory in a computer-readable storage medium. For example, the memory may be random access memory (RAM), read-only memory (ROM), or flash memory, or any combination thereof, configured to store programs executed by a processor. The memory may include at least one memory chip. Here, the embodiments of this disclosure do not limit this.

[0200] In one exemplary embodiment, the bus may include, in addition to a data bus, a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, in... Figure 9 In this embodiment, various buses are labeled as bus 93. This disclosure does not limit the scope of the embodiments.

[0201] In implementation, the processing performed by the computer device can be accomplished through integrated logic circuits in the processor's hardware or through software instructions. That is, the method steps of this disclosure can be executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0202] In one exemplary embodiment, the computer device may include various forms of digital computers, such as educational tablets, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Alternatively, the computer device may also include various forms of mobile electronic devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0203] Based on the same inventive concept, embodiments of this disclosure also provide a non-transient computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to perform the steps of the video question-and-answer method in one or more of the above exemplary embodiments.

[0204] In one exemplary embodiment, the computer-readable storage medium described above may include, but is not limited to, ROM / RAM, magnetic disks, or optical disks. This disclosure does not limit the scope of the embodiment.

[0205] The descriptions of the computer device or computer-readable storage medium embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the computer device or computer-readable storage medium embodiments of this disclosure, those skilled in the art should refer to the descriptions of the method embodiments of this disclosure for understanding. Further details will not be repeated here.

[0206] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0207] While the embodiments disclosed herein are as described above, the above content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit this disclosure. Any person skilled in the art to which this disclosure pertains may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed herein, but the scope of patent protection of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. A video question-answering method, characterized in that, include: Obtaining the question to be answered and the video to be identified includes: performing semantic understanding on the question to be answered to obtain the question category; using the question category as the video category to be identified, determining multiple candidate videos whose video category is the video category to be identified from a pre-set database, wherein the candidate videos with the number of matched subtitle texts greater than a preset threshold are determined as the video to be identified; or, according to the number of matched subtitle texts corresponding to each of the multiple candidate videos, sorting the multiple candidate videos in descending order, and determining the candidate video with the highest ranking as the video to be identified; Based on the question to be answered, from the video to be identified, an initial answer video segment containing subtitle text that matches the question to be answered and its adjacent video segments to be identified that do not contain subtitle text are segmented. Based on the video segment to be identified, the matched subtitle text, and the initial response video segment, predict whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment, and obtain a first prediction result indicating that there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment, or a second prediction result indicating that there is no semantic similarity between the content of the video segment to be identified and the content of the initial response video segment. Based on whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment, it is determined whether the final response video segment contains the video segment to be identified. Specifically, when the obtained prediction result is the first prediction result, the initial response video segment and the video segment to be identified are merged, and the merged video segment is determined as the final response video segment; or, when the obtained prediction result is the second prediction result, it is determined that the final response video segment does not contain the video segment to be identified, and the initial response video segment is determined as the final response video segment.

2. The video question-answering method according to claim 1, characterized in that, The step of predicting whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment, based on the video segment to be identified, the matched subtitle text, and the initial response video segment, includes: Based on the video segment to be identified, the matched subtitle text, and the initial response video segment, a feature vector to be predicted is constructed; Based on the feature vector to be predicted, predict whether there is semantic similarity between the content of the video segment to be identified and the content of the initial response video segment.

3. The video question-answering method according to claim 2, characterized in that, The step of constructing a feature vector to be predicted based on the video segment to be identified, the matched subtitle text, and the initial response video segment includes: Feature extraction is performed on the video segment to be identified to obtain a first video feature vector; Feature extraction is performed on the matched subtitle text to obtain a text feature vector; Using an attention mechanism, the first video feature vector and the text feature vector are modally fused to obtain a feature vector incorporating semantics; Feature extraction is performed on the initial response video segment to obtain a second video feature vector; The second video feature vector and the semantically integrated feature vector are concatenated to obtain the feature vector to be predicted.

4. The video question-answering method according to claim 2 or 3, characterized in that, The step of predicting whether the content of the video segment to be identified and the content of the initial response video segment have semantic similarity based on the feature vector to be predicted includes: inputting the feature vector to be predicted into a first natural language model to obtain the first prediction result or the second prediction result.

5. The video question-answering method according to claim 1, characterized in that, The process of obtaining the question to be answered and the video to be identified includes: in response to a query operation, obtaining the question to be answered; and based on the question to be answered, obtaining the video to be identified and the subtitle file corresponding to the video to be identified from a pre-set database. The step of determining, based on the question to be answered, an initial answer video segment containing subtitle text matching the question to be answered and its adjacent video segments to be identified that do not contain subtitle text from the video to be identified includes: matching the question to be answered with the subtitle file corresponding to the video to be identified to obtain the matching subtitle text; and segmenting the initial answer video segment and the video segments to be identified from the video to be identified based on the time information of the matching subtitle text.

6. The video question-answering method according to claim 5, characterized in that, The step of matching the question to be answered with the subtitle file corresponding to the video to be identified to obtain the matched subtitle text includes: For each subtitle text in the subtitle file corresponding to the video to be identified, perform the following operations: Configure a preset prompt text template for the question to be answered and the subtitle text to construct the text to be judged; Based on the text to be judged, a second natural language model is used to predict whether the subtitle text is the answer to the question to be answered. When the caption text is an answer that matches the question to be answered, the caption text is used as the matched caption text.

7. The video question-answering method according to claim 6, characterized in that, The step of predicting whether the subtitle text is an answer to the question to be answered, based on the text to be judged and using a second natural language model, includes: The text to be judged is input into the second natural language model to obtain the output probability value, which is used to indicate the magnitude of the semantic relevance between the question to be answered and the subtitle text; Based on whether the probability value is greater than a preset probability threshold, it is predicted whether the subtitle text is the answer to the question to be answered.

8. The video question-answering method according to claim 6, characterized in that, The process of configuring a preset prompt text template for the question to be answered and the subtitle text to construct the text to be judged includes: Insert the question to be answered into the preset prompt text template to obtain the text to be concatenated; The text to be concatenated is concatenated with the subtitle text to obtain the text to be judged.

9. The video question-answering method according to claim 5, characterized in that, The step of retrieving the video to be identified and its corresponding subtitle file from a pre-set database based on the question to be answered includes: Perform semantic understanding on the question to be answered to obtain the question category; Using the problem category as the video category to be identified, multiple candidate videos with the video category to be identified and their corresponding subtitle files are determined from the pre-set database. The video to be identified and its corresponding subtitle file are determined from the plurality of candidate videos and their respective subtitle files.

10. The video question-answering method according to claim 1, characterized in that, The method further includes: Obtain the video from which subtitles are to be extracted; The video from which the subtitles are to be extracted is subjected to frame extraction processing to obtain an image frame set corresponding to the video from which the subtitles are to be extracted, the image frame set including multiple keyframes; For each keyframe in the image frame set, a subtitle extraction operation is performed to obtain the subtitle file corresponding to the video from which the subtitle to be extracted is obtained. The subtitle extraction operation includes: performing optical character recognition processing on the keyframe to obtain the coordinates of multiple text boxes and multiple texts in the keyframe; and determining the subtitle text of the keyframe based on the coordinates of the multiple text boxes and the multiple texts in the keyframe. The subtitle file corresponding to the video from which the subtitles are to be extracted is associated with the video from which the subtitles are to be extracted and stored in the database.

11. The video question-answering method according to claim 10, characterized in that, The process of determining the caption text of a keyframe based on the coordinates of multiple text boxes and multiple texts within the keyframe includes: From multiple texts in keyframes adjacent to this keyframe, determine the contrast text corresponding to the position of each text in this keyframe; Calculate the text difference between each text in the keyframe and its corresponding contrast text; Based on the coordinates of multiple text boxes in the keyframe, determine the frequency of the coordinates of the text box to which each text belongs in the keyframe appears. Based on the text difference between each text in the keyframe and its corresponding comparison text, as well as the frequency of the coordinates of the text box to which each text in the keyframe belongs, the multiple texts in the keyframe are sorted. Based on the sorting results, the caption text of the keyframe is determined from multiple texts in the keyframe.

12. The video question-answering method according to claim 10, characterized in that, The step of performing subtitle extraction on each keyframe of the image frame set to obtain the subtitle file corresponding to the video from which the subtitles to be extracted includes: The text detection algorithm determines the text box information corresponding to each of the multiple keyframes in the image frame set. The text box information includes at least one of the following: the number of text boxes and the coordinates of the text boxes. Based on the text box information corresponding to each of the multiple keyframes in the image frame set, keyframes that meet preset conditions are removed from the image frame set to obtain a processed image frame set; wherein, the keyframes that meet the preset conditions include at least one of the following: a first type of keyframe with a text box number greater than a preset threshold, a second type of keyframe with text box coordinates outside a preset range, and a third type of keyframe with no text box. The subtitle extraction operation is performed on each keyframe in the processed image frame set to obtain the subtitle file corresponding to the video from which the subtitle to be extracted.

13. The video question-answering method according to claim 1, characterized in that, The method further includes: pushing the final answer video clip, or, based on the start and end times of the final answer video clip, pushing a video to be identified carrying identification information indicating the location of the final answer video clip.

14. A computer device, characterized in that, include: The processor and a memory storing a computer program executable on the processor, wherein the processor is configured to implement the steps of the video question-answering method as described in any one of claims 1 to 13 when executing the program.

15. A non-transient computer-readable storage medium, characterized in that, The device stores computer-executable instructions, wherein the computer-executable instructions are configured to perform the steps of the video question-answering method as described in any one of claims 1 to 13.

Images