Video question and answer method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video question answering method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of performing information fusion processing on multiple pieces of teaching key text information, second association time information of each piece of teaching key text information, multiple pieces of teaching courseware text information and third association time information of each piece of teaching courseware text information, obtaining a plurality of pieces of teaching text information and fourth associated time information corresponding to each piece of teaching text information; storing the multiple pieces of teaching text information corresponding to each teaching video and the fourth associated time information into a retrieval engine; receiving problem information sent by the user side; according to the retrieval content, obtaining corresponding target teaching text information and fourth associated time information from a retrieval engine; and returning a corresponding reply result to the user side according to the target teaching text information and the fourth association time information. The application range is wide, the accuracy and continuity of reply results are improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video question-and-answer method, device, electronic device, storage medium, and program product. Background Art

[0002] With the continuous development of new media technology, a large number of teaching videos have emerged on the Internet. However, due to the abundance of teaching videos, users spend a lot of time selecting and searching for teaching videos. Video Q&A is an interactive multimedia technology that combines video content and Q&A interaction, aiming to automatically answer questions raised by users based on the video content. Video Q&A can help users efficiently extract and understand information from videos.

[0003] Currently, video question-answering mainly involves dividing the full video into segments with and without subtitles, and then giving answers to questions based on the subtitles.

[0004] However, the existing technology mainly relies on subtitles, has a small scope of application, and the accuracy and coherence of question answers are low, which affects the user experience. Summary of the invention

[0005] The present application provides a video question-and-answer method, device, electronic device, storage medium and program product to solve the technical problems that the prior art relies on subtitles, has a small scope of application, has low accuracy and coherence in answering questions, and affects user experience.

[0006] In a first aspect, the present application provides a video question-answering method, comprising:

[0007] Get a preset number of instructional videos;

[0008] For each teaching video, audio extraction processing is performed to obtain teaching audio;

[0009] The teaching audio is transcribed to obtain a plurality of teaching content text information and first associated time information of each teaching content text information;

[0010] Input the plurality of teaching content text information and the first associated time information of each teaching content text information into a preset large language model to output a plurality of teaching key text information and the second associated time information of each teaching key text information;

[0011] For each teaching video, frame extraction is performed according to a preset frame extraction frequency to obtain a plurality of teaching pictures;

[0012] Performing text recognition on the multiple teaching pictures to obtain multiple teaching courseware text information and third associated time information of each teaching courseware text information;

[0013] Performing information fusion processing on the plurality of teaching key text information and the second associated time information of each teaching key text information and the plurality of teaching courseware text information and the third associated time information of each teaching courseware text information to obtain the plurality of teaching text information and the fourth associated time information corresponding to each teaching text information;

[0014] storing the plurality of teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information in the search engine;

[0015] Receiving question information sent by a user, wherein the question information includes search content;

[0016] According to the search content, obtaining corresponding target teaching text information and fourth associated time information corresponding to the target teaching text information from the search engine;

[0017] According to the target teaching text information and the fourth associated time information corresponding to the target teaching text information, a corresponding reply result is returned to the user terminal.

[0018] Optionally, according to the method as described above, the information fusion processing of the multiple teaching key text information and the second associated time information of each teaching key text information and the multiple teaching courseware text information and the third associated time information of each teaching courseware text information is performed to obtain the multiple teaching text information and the fourth associated time information of each teaching text information, including: determining the main timeline according to the third associated time information of each teaching courseware text information; aligning the multiple teaching key text information and the multiple teaching courseware text information on the main timeline according to the second associated time information of each teaching key text information and the third associated time information of each teaching courseware text information to obtain the multiple synchronous text information and the synchronous associated time information of each synchronous text information; correcting the synchronous associated time information of the multiple synchronous text information and each synchronous text information to obtain the fourth associated time information corresponding to the multiple teaching text information and each teaching text information.

[0019] Optionally, in the method as described above, each of the multiple synchronized text messages includes a teaching key text message and a teaching courseware text message synchronized with the teaching key text message; accordingly, the synchronization-associated time information of the multiple synchronized text messages and each synchronized text message is corrected to obtain multiple teaching text messages and a fourth associated time information corresponding to each teaching text message, including: traversing the multiple synchronized text messages in sequence according to the time sequence of the synchronization-associated time information, and executing the following steps for the synchronized text messages traversed: obtaining the teaching key text information and the teaching courseware text information in the synchronized text information; calculating the teaching key text The first correlation between the information and the teaching courseware text information; if it is determined that the first correlation does not reach the preset first threshold, the start time of the synchronization associated time information of the synchronization text information is obtained; the second correlation between the teaching content text information and the teaching courseware text information within the preset time period before and after the start time is calculated; if it is determined that the second correlation reaches the preset second threshold, the teaching key text information in the synchronization text information is corrected to the teaching content text information or the teaching courseware text information within the preset time period before and after the start time; the teaching text information is obtained according to the corrected synchronization text information; the synchronization associated time information is determined to be the fourth associated time information of the teaching text information.

[0020] Optionally, the method as described above, returning a corresponding reply result to the user terminal based on the target teaching text information and the fourth associated time information corresponding to the target teaching text information, includes: inputting the search content and the target teaching text information and the fourth associated time information corresponding to the target teaching text information into a preset big model to determine whether the third correlation between the target teaching text information and the search content reaches a preset third threshold; if it is determined that the third correlation reaches the preset third threshold, then using the preset big model to generate a corresponding reply result based on the question information and the target teaching text information and the fourth associated time information corresponding to the target teaching text information.

[0021] Optionally, the method as described above, after determining whether the third correlation between the target teaching text information and the search content reaches a preset third threshold, further includes: if it is determined that the third correlation does not reach the preset third threshold, then using the preset large model to generate a corresponding reply result based on the question information.

[0022] Optionally, as described above, after receiving the question information sent by the user terminal, the method further includes: inputting the question information into a preset large model to determine whether the question information is a content query or a video search; if the question information is determined to be a content query or a video search, executing the step of "according to the search content, obtaining the corresponding target teaching text information and the fourth associated time information corresponding to the target teaching text information from the search engine."

[0023] In a second aspect, the present application provides a video question-and-answer device, comprising:

[0024] An acquisition module, used to acquire a preset number of teaching videos;

[0025] The extraction module is used to perform audio extraction processing on each teaching video to obtain teaching audio;

[0026] An identification module is used to transcribe the teaching audio to obtain a plurality of teaching content text information and first associated time information of each teaching content text information;

[0027] An input module, used for inputting the plurality of teaching content text information and the first associated time information of each teaching content text information into a preset large language model, so as to output a plurality of teaching key text information and the second associated time information of each teaching key text information;

[0028] The extraction module is further used to perform frame extraction processing on each teaching video according to a preset frame extraction frequency to obtain multiple teaching pictures;

[0029] The recognition module is further used to perform text recognition on the multiple teaching pictures to obtain multiple teaching courseware text information and third associated time information of each teaching courseware text information;

[0030] A fusion module, used for performing information fusion processing on the plurality of teaching key text information and the second associated time information of each teaching key text information and the plurality of teaching courseware text information and the third associated time information of each teaching courseware text information, so as to obtain the fourth associated time information corresponding to the plurality of teaching text information and each teaching text information;

[0031] A storage module, used for storing the multiple teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information into the search engine;

[0032] A receiving module, used for receiving question information sent by a user terminal, wherein the question information includes search content;

[0033] The acquisition module is further used to acquire corresponding target teaching text information and fourth associated time information corresponding to the target teaching text information from the search engine according to the search content;

[0034] The sending module is used to return the corresponding reply result to the user terminal according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information.

[0035] In a third aspect, the present application provides a service device, comprising: a processor, and a memory communicatively connected to the processor;

[0036] The memory stores computer-executable instructions;

[0037] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video question-and-answer method as described in the first aspect and various possible designs of the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect and various possible designs of the first aspect.

[0039] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect and various possible designs of the first aspect.

[0040] The video question-and-answer method, device, electronic device, storage medium and program product provided by the present application transcribe the teaching audio to obtain multiple teaching content text information and the first associated time information of each teaching content text information; input the multiple teaching content text information and the first associated time information of each teaching content text information into a preset large language model to output multiple teaching key text information and the second associated time information of each teaching key text information; perform frame extraction processing on each teaching video to obtain multiple teaching pictures; perform text recognition on multiple teaching pictures to obtain multiple teaching courseware text information and the third associated time information of each teaching courseware text information; multiple teaching key text information and The second associated time information of each key teaching text information and multiple teaching courseware text information and the third associated time information of each teaching courseware text information are processed for information fusion to obtain multiple teaching text information and the fourth associated time information corresponding to each teaching text information; multiple teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information are stored in the search engine; the search content sent by the user end is received; according to the search content, the corresponding target teaching text information and the fourth associated time information corresponding to the target teaching text information are obtained from the search engine; according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information, the corresponding reply result is returned to the user end. The application scope is wide, the accuracy and consistency of the reply result are improved, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 A schematic diagram of a scenario of a video question-answering method provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of a video question-and-answer method provided by one embodiment of the present application;

[0044] Figure 3 A schematic diagram of a process for obtaining and storing teaching text information and fourth associated time information provided by an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of a video question-and-answer device provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the hardware structure of the service device provided in an embodiment of the present application.

[0047] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0050] Currently, the video question-answering with large models mainly relies on subtitles. The complete video is divided into segments with and without subtitles. Then, based on the similarity between subtitles and questions, segments with relevant subtitles and adjacent segments are obtained, and the relevance of the segments is calculated, so that multiple segments are given to answer questions. On the one hand, this method is overly dependent on subtitles. Videos without subtitles cannot use this method, and its scope of application is small. On the other hand, it cannot avoid the problem of spoken language in the video or subtitle information, resulting in less effective information in the subtitles. The subtitle segments are not necessarily semantic, so when making relevance judgments, they may be inaccurate, resulting in low accuracy and coherence in answering questions, affecting the user experience.

[0051] In order to solve the above technical problems, the embodiments of the present application propose the following technical ideas: Considering that the application scope of video questions and answers that rely on subtitles is small, and when making relevance judgments, they may be inaccurate, resulting in low accuracy and coherence in answering questions, affecting the user experience. The inventors have come up with a method of extracting and summarizing the text information of the teaching content transcribed from the teaching audio through a large model, and fusing the obtained key teaching text information with the teaching courseware text information in time sequence. This further reduces the error rate of speech recognition and text recognition, has a wide scope of application, improves the accuracy and coherence of question answers, and improves the user experience.

[0052] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0053] Figure 1 Schematic diagram of a scenario of a video question-answering method provided in an embodiment of the present application. Figure 1 As shown, the scenario provided in this embodiment includes a user terminal 101 and a service device 102 .

[0054] The user terminal 101 may be a mobile phone or a computer.

[0055] The service device 102 may be a server. Optionally, it may be one server or a cluster consisting of multiple servers.

[0056] Specifically, the user terminal 101 sends a question message to the service device 102; the service device 102 obtains the corresponding target teaching text information and the fourth associated time information corresponding to the target teaching text information according to the search content of the question message; the service device 102 returns the corresponding reply result to the user terminal 101 according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information.

[0057] Figure 2 A flowchart of a video question-answering method provided by an embodiment of the present application. The execution subject of this embodiment may be Figure 1 The service device 102 shown in FIG. 1 may also be other computer devices, which is not particularly limited in this embodiment. Figure 2 As shown, the method includes:

[0058] S201: Obtain a preset number of teaching videos.

[0059] Specifically, a preset number of teaching videos sent by the receiving management terminal are received.

[0060] S202: For each teaching video, perform audio extraction processing to obtain teaching audio.

[0061] Specifically, a speech separation tool is used to extract the teaching audio from the teaching video.

[0062] The speech separation tool may be a third-party toolkit or a ffmpeg (Fast-Forward-Moving-Picture Expert Group) tool.

[0063] S203: Transcribe the teaching audio to obtain multiple teaching content text information and the first associated time information of each teaching content text information.

[0064] Specifically, a speech recognition algorithm is used to transcribe the teaching audio to obtain multiple teaching content text information and the first associated time information of each teaching content text information.

[0065] The speech recognition algorithm may be ASR (Automatic Speech Recognition).

[0066] The format of the multiple teaching content text information and the first associated time information of each teaching content text information may be JSON.

[0067] S204: Input multiple teaching content text information and the first associated time information of each teaching content text information into a preset large language model to output multiple teaching key text information and the second associated time information of each teaching key text information.

[0068] Among them, the preset large language model can be a LoRA (Low-Rank Adaptation) model fine-tuned based on the general large language model.

[0069] Among them, the general large language model can be trained based on Llama (Large Language Model Meta AI), Yi or Chatglm (Chat General Language Model, general chat language model) and the like.

[0070] It should be noted that the LoRA model is obtained by fine-tuning the general large language model using a manually annotated dataset.

[0071] The format of the multiple pieces of key teaching text information and the second associated time information of each piece of key teaching text information may be JSON.

[0072] S205: For each teaching video, frame extraction is performed according to a preset frame extraction frequency to obtain a plurality of teaching pictures.

[0073] Specifically, a frame extraction algorithm is used to perform frame extraction processing on each teaching video according to a preset frame extraction frequency to obtain multiple teaching pictures.

[0074] Among them, the frame extraction algorithm can be fixed interval frame extraction, key frame extraction, etc.

[0075] The preset frame extraction frequency may be one frame per second.

[0076] S206: Perform text recognition on the multiple teaching pictures to obtain multiple teaching courseware text information and third associated time information of each teaching courseware text information.

[0077] Specifically, a text recognition algorithm is used to perform text recognition on multiple teaching pictures to obtain multiple teaching courseware text information and the third associated time information of each teaching courseware text information.

[0078] The text recognition algorithm may be OCR (Optical Character Recognition).

[0079] Among them, the format of the multiple teaching courseware text information and the third associated time information of each teaching courseware text information can be JSON.

[0080] S207: Perform information fusion processing on multiple teaching key text information and the second associated time information of each teaching key text information and multiple teaching courseware text information and the third associated time information of each teaching courseware text information to obtain multiple teaching text information and the fourth associated time information corresponding to each teaching text information.

[0081] Specifically, S207 specifically includes S2071 to S2073:

[0082] S2071: Determine the main timeline based on the third associated time information of each teaching courseware text information.

[0083] Specifically, the third associated time information of each teaching courseware text information is sorted in chronological order, and the main timeline is determined according to the sorted third associated time information.

[0084] S2072: According to the second associated time information of each teaching key text information and the third associated time information of each teaching courseware text information, multiple teaching key text information and multiple teaching courseware text information are aligned on the main timeline to obtain multiple synchronous text information and synchronous associated time information of each synchronous text information.

[0085] Specifically, the second associated time information of each teaching key text information and the third associated time information of each teaching courseware text information are aligned on the main timeline. According to the aligned second associated time information and third associated time information, multiple teaching key text information and multiple teaching courseware text information are aligned on the main timeline to obtain multiple synchronized text information and the synchronized associated time information of each synchronized text information.

[0086] S2073: Correct the multiple synchronous text information and the synchronous associated time information of each synchronous text information to obtain the multiple teaching text information and the fourth associated time information corresponding to each teaching text information.

[0087] Specifically, a semantic consistency check is performed on the synchronized teaching key text information and teaching courseware text information in each synchronized text information, and the teaching key text information that fails the check is marked as a conflict point; if the check result is failure, the context of the conflict point is analyzed to determine whether the teaching key text information and the teaching courseware text information are complementary; if it is determined that the teaching key text information and the teaching courseware text information are complementary, the teaching key text information and the teaching courseware text information are merged to obtain the teaching text information; the time of the conflict point is determined to be the fourth associated time information corresponding to the teaching text information.

[0088] S208: storing the multiple teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information in the search engine.

[0089] Specifically, the multiple teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information are input into the vectorization model to output the vectorized multiple teaching text information and the fourth associated time information of each teaching text information; the vectorized multiple teaching text information and the fourth associated time information of each teaching text information are stored in the retrieval engine.

[0090] Among them, the vectorized model can be BGE-M3, M3E, etc.

[0091] Figure 3 A schematic diagram of the process of obtaining and storing teaching text information and fourth associated time information provided in one embodiment of the present application.

[0092] S209: receiving question information sent by the user, wherein the question information includes search content.

[0093] S210: Acquire corresponding target teaching text information and fourth associated time information corresponding to the target teaching text information from the search engine according to the search content.

[0094] Specifically, keyword extraction is performed on the search content to obtain search keywords; the search keywords are vectorized to obtain vectorized keywords; and a query is performed in the search engine based on the vectorized keywords to obtain target teaching text information and fourth associated time information corresponding to the target teaching text information.

[0095] S211: Return a corresponding reply result to the user terminal according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information.

[0096] Specifically, S211 includes S2111 to S2112:

[0097] S2111: Input the search content, target teaching text information and the fourth associated time information corresponding to the target teaching text information into a preset large model to determine whether the third correlation between the target teaching text information and the search content reaches a preset third threshold.

[0098] Among them, the target teaching text information may include one or more of the following: target teaching key text information, target teaching courseware text information, target teaching content text information, etc.

[0099] Specifically, the search content, target teaching text information and the fourth associated time information corresponding to the target teaching text information are input into a preset large model to calculate the cosine correlation between the target teaching text information and the search content; the preset large model is used to determine whether the cosine correlation reaches a preset third threshold.

[0100] S2112: If it is determined that the third correlation reaches the preset third threshold, a preset large model is used to generate a corresponding reply result according to the question information, the target teaching text information, and the fourth correlation time information corresponding to the target teaching text information.

[0101] Specifically, a preset large model is used to analyze and summarize the target teaching text information according to the question information to obtain the reply content; and a reply result corresponding to the question information is generated according to the reply content and the fourth associated time information corresponding to the target teaching text information.

[0102] Among them, the reply result includes one or more of the following: reply content, fourth associated time information corresponding to the target teaching text information, video clips corresponding to the target teaching text information, etc.

[0103] From the above description, it can be seen that the present application transcribes the teaching audio to obtain a plurality of teaching content text information and the first associated time information of each teaching content text information; inputs the plurality of teaching content text information and the first associated time information of each teaching content text information into a preset large language model to output a plurality of teaching key text information and the second associated time information of each teaching key text information; performs frame extraction processing on each teaching video to obtain a plurality of teaching pictures; performs text recognition on a plurality of teaching pictures to obtain a plurality of teaching courseware text information and the third associated time information of each teaching courseware text information; performs text recognition on a plurality of teaching key text information and the third associated time information of each teaching courseware text information; performs frame extraction processing on a plurality of teaching videos to obtain a plurality of teaching key text information and the third associated time information of each teaching key text information; performs frame extraction processing on a plurality of teaching pictures to obtain a plurality of teaching courseware text information and the third associated time information of each teaching courseware text information; performs frame extraction processing on a plurality of teaching key text information and the third associated time information of each teaching key text information. The second associated time information and multiple teaching courseware text information and the third associated time information of each teaching courseware text information are processed for information fusion to obtain multiple teaching text information and the fourth associated time information corresponding to each teaching text information; the multiple teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information are stored in the search engine; the search content sent by the user end is received; according to the search content, the corresponding target teaching text information and the fourth associated time information corresponding to the target teaching text information are obtained from the search engine; according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information, the corresponding reply result is returned to the user end. The application scope is wide, the accuracy and consistency of the reply result are improved, and the user experience is improved.

[0104] In one embodiment of the present application, based on the above embodiment, each of the multiple synchronous text messages includes a teaching key text message and a teaching courseware text message synchronized with the teaching key text message. Accordingly, the specific process of step S2073 is described in detail as follows:

[0105] Sa: According to the time sequence of the synchronization-related time information, traverse the multiple synchronization text information in sequence, and execute the following steps Sa-1 to Sa-7 for the traversed synchronization text information.

[0106] Sa-1: Obtain key teaching text information and teaching courseware text information in the synchronized text information.

[0107] Specifically, the key teaching text information and teaching courseware text information in the synchronous text information are extracted.

[0108] Sa-2: Calculate the first correlation between the key teaching text information and the teaching courseware text information.

[0109] The first relevance may be a semantic relevance.

[0110] Specifically, text cleaning and word segmentation are performed on the teaching key text information and the teaching courseware text information; feature extraction is performed on the teaching key text information and the teaching courseware text information after word segmentation to obtain the teaching key text information vector and the teaching courseware text information vector; the cosine correlation between the teaching key text information vector and the teaching courseware text information vector is calculated to obtain the first correlation.

[0111] Sa-3: If it is determined that the first correlation does not reach the preset first threshold, the start time of the synchronization associated time information of the synchronization text information is obtained.

[0112] Specifically, the attributes of the synchronization-related time information are parsed; and the start time in the attributes is identified and obtained.

[0113] Sa-4: Calculate the second correlation between the teaching content text information and the teaching courseware text information within a preset time period before and after the start time.

[0114] The second relevance may be a semantic relevance.

[0115] Specifically, text cleaning and word segmentation are performed on the teaching content text information and teaching courseware text information within a preset time period before and after the start time; feature extraction is performed on the teaching content text information and teaching courseware text information within a preset time period before and after the start time after word segmentation to obtain a teaching content text information vector and a teaching courseware text information vector; and the cosine correlation between the teaching content text information vector and the teaching courseware text information vector is calculated to obtain a second correlation.

[0116] Sa-5: If it is determined that the second correlation reaches the preset second threshold, the key teaching text information in the correction synchronization text information is the teaching content text information or teaching courseware text information within a preset time period before and after the start time.

[0117] Specifically, if it is determined that the second correlation reaches a preset second threshold, the key teaching text information in the synchronized text information is updated to be the teaching content text information or teaching courseware text information within a preset time period before and after the start time.

[0118] Sa-6: Obtain teaching text information based on the corrected synchronized text information.

[0119] Specifically, the teaching text information is generated based on the corrected synchronization text information and the teaching content text information corresponding to the fourth associated time information of the teaching text information.

[0120] Sa-7: Determine the synchronization-related time information as the fourth-related time information of the teaching text information.

[0121] As can be seen from the above description, the present application calculates the first correlation between the key teaching text information and the teaching courseware text information; if it is determined that the first correlation does not reach the preset first threshold, then obtains the start time of the synchronization association time information of the synchronization text information; calculates the second correlation between the teaching content text information and the teaching courseware text information in the preset time period before and after the start time; if it is determined that the second correlation reaches the preset second threshold, then corrects the key teaching text information in the synchronization text information to the teaching content text information or teaching courseware text information in the preset time period before and after the start time. This further reduces the error rate of speech recognition and text recognition, improves the accuracy and coherence of the response results, and improves the user experience.

[0122] In one embodiment of the present application, based on the above embodiment, after step S2111, a process of returning a corresponding reply result using the dialogue function of the large model is further included, as described in detail as follows:

[0123] S2113: If it is determined that the third correlation does not reach the preset third threshold, a preset large model is used to generate a corresponding reply result according to the question information.

[0124] From the above description, it can be seen that when the relevant answer cannot be retrieved, the application uses the dialogue function of the large model to answer the user's question, reduce the user's waiting time, and improve the user experience.

[0125] In one embodiment of the present application, based on the above embodiment, after step S209, a process of determining whether the user question information is a content query or a video search is further included, as described in detail as follows:

[0126] S212: Input the question information into a preset macro model to determine whether the question information is a content query or a video search.

[0127] S213: If the question information is determined to be content query or video search, execute step S210.

[0128] In one embodiment of the present application, if it is determined that the question information is not a content query or a video search, a prompt indicating that the question input is incorrect is returned to the user end, so that the user of the user end re-enters the question information.

[0129] Figure 4 This is a schematic diagram of the structure of the video question-answering device provided in the embodiment of the present application. Figure 4 As shown, the video question-answering device 40 includes: an acquisition module 401, an extraction module 402, a recognition module 403, an input module 404, a fusion module 405, a storage module 406, a receiving module 407 and a sending module 408.

[0130] An acquisition module 401 is used to acquire a preset number of teaching videos;

[0131] The extraction module 402 is used to perform audio extraction processing on each teaching video to obtain teaching audio;

[0132] The recognition module 403 is used to transcribe the teaching audio to obtain a plurality of teaching content text information and the first associated time information of each teaching content text information;

[0133] An input module 404 is used to input the plurality of teaching content text information and the first associated time information of each teaching content text information into a preset large language model to output a plurality of teaching key text information and the second associated time information of each teaching key text information;

[0134] The extraction module 402 is further used to perform frame extraction processing on each teaching video according to a preset frame extraction frequency to obtain multiple teaching pictures;

[0135] The recognition module 403 is further used to perform text recognition on the multiple teaching pictures to obtain multiple teaching courseware text information and third associated time information of each teaching courseware text information;

[0136] The fusion module 405 is used to perform information fusion processing on the plurality of teaching key text information and the second associated time information of each teaching key text information and the plurality of teaching courseware text information and the third associated time information of each teaching courseware text information, so as to obtain the fourth associated time information corresponding to the plurality of teaching text information and each teaching text information;

[0137] Storage module 406, used for storing the multiple teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information into the search engine;

[0138] Receiving module 407, used to receive question information sent by the user end, wherein the question information includes search content;

[0139] The acquisition module 401 is further used to acquire corresponding target teaching text information and fourth associated time information corresponding to the target teaching text information from the search engine according to the search content;

[0140] The sending module 408 is used to return a corresponding reply result to the user terminal according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information.

[0141] In a possible design, the fusion module 405 is specifically used to: determine the main timeline according to the third associated time information of each teaching courseware text information; align the multiple teaching key text information and the multiple teaching courseware text information on the main timeline according to the second associated time information of each teaching key text information and the third associated time information of each teaching courseware text information to obtain multiple synchronized text information and the synchronized associated time information of each synchronized text information; correct the multiple synchronized text information and the synchronized associated time information of each synchronized text information to obtain multiple teaching text information and the fourth associated time information corresponding to each teaching text information.

[0142] In a possible design, each of the multiple synchronized text information includes a teaching key text information and a teaching courseware text information synchronized with the teaching key text information; accordingly, the fusion module 405, when "correcting the multiple synchronized text information and the synchronization associated time information of each synchronized text information to obtain multiple teaching text information and the fourth associated time information corresponding to each teaching text information", is specifically used to: traverse the multiple synchronized text information in turn according to the time sequence of the synchronization associated time information, and for the synchronized text information traversed, perform the following steps: obtain the teaching key text information and the teaching courseware text information in the synchronized text information; calculate the A first correlation between the teaching key text information and the teaching courseware text information; if it is determined that the first correlation does not reach the preset first threshold, then obtaining the start time of the synchronization associated time information of the synchronization text information; calculating the second correlation between the teaching content text information and the teaching courseware text information within the preset time period before and after the start time; if it is determined that the second correlation reaches the preset second threshold, then correcting the teaching key text information in the synchronization text information to the teaching content text information or the teaching courseware text information within the preset time period before and after the start time; obtaining the teaching text information based on the corrected synchronization text information; determining the synchronization associated time information to be the fourth associated time information of the teaching text information.

[0143] In one possible design, the sending module 408 is specifically used to: input the search content and the target teaching text information as well as the fourth associated time information corresponding to the target teaching text information into a preset big model to determine whether the third correlation between the target teaching text information and the search content reaches a preset third threshold; if it is determined that the third correlation reaches the preset third threshold, the preset big model is used to generate a corresponding reply result according to the question information and the target teaching text information as well as the fourth associated time information corresponding to the target teaching text information.

[0144] In a possible design, the sending module 408 is further specifically used to: if it is determined that the third correlation does not reach a preset third threshold, then use the preset large model to generate a corresponding reply result according to the question information.

[0145] In a possible design, the receiving module 407 is also specifically used to: input the question information into a preset large model to determine whether the question information is a content query or a video search; if the question information is determined to be a content query or a video search, execute the step of "according to the search content, obtaining the corresponding target teaching text information and the fourth associated time information corresponding to the target teaching text information from the search engine."

[0146] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0147] Figure 5 Schematic diagram of the hardware structure of the service device provided in the embodiment of the present application. Figure 5 As shown, the service device 50 of this embodiment includes: at least one processor 501 and a memory 502; the memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, so that at least one processor executes the above video question-answering method;

[0148] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0149] When the memory 502 is independently provided, the service device further includes a bus 503 for connecting the memory 502 and the processor 501 .

[0150] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video question-and-answer method as described above is implemented.

[0151] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the video question-and-answer method as described above.

[0152] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0153] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0154] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0155] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0156] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not otherwise specified, the processor may be any appropriate hardware processor, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), and an ASIC (Application-Specific Integrated Circuit). If not otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as RRAM (Resistive Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random-Access Memory), EDRAM (Enhanced Dynamic Random Access Memory), HBM (High-Bandwidth Memory), HMC (Hybrid Memory Cube), etc.

[0157] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned memory includes: U disk, ROM (Read-Only Memory), RAM (Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program codes.

[0158] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0160] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video question-answering method, characterized in that: Applied to service equipment, including: Get a preset number of instructional videos; For each teaching video, audio extraction processing is performed to obtain teaching audio; The teaching audio is transcribed to obtain a plurality of teaching content text information and first associated time information of each teaching content text information; Input the plurality of teaching content text information and the first associated time information of each teaching content text information into a preset large language model to output a plurality of teaching key text information and the second associated time information of each teaching key text information; For each teaching video, frame extraction is performed according to a preset frame extraction frequency to obtain a plurality of teaching pictures; Performing text recognition on the multiple teaching pictures to obtain multiple teaching courseware text information and third associated time information of each teaching courseware text information; Performing information fusion processing on the plurality of teaching key text information and the second associated time information of each teaching key text information and the plurality of teaching courseware text information and the third associated time information of each teaching courseware text information to obtain the plurality of teaching text information and the fourth associated time information corresponding to each teaching text information; storing the plurality of teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information in the search engine; Receiving question information sent by a user, wherein the question information includes search content; According to the search content, obtaining corresponding target teaching text information and fourth associated time information corresponding to the target teaching text information from the search engine; According to the target teaching text information and the fourth associated time information corresponding to the target teaching text information, a corresponding reply result is returned to the user terminal.

2. The method according to claim 1, characterized in that The information fusion processing is performed on the plurality of teaching key text information and the second associated time information of each teaching key text information and the plurality of teaching courseware text information and the third associated time information of each teaching courseware text information to obtain the plurality of teaching text information and the fourth associated time information of each teaching text information, including: Determine the main timeline according to the third associated time information of each teaching courseware text information; According to the second associated time information of each teaching key text information and the third associated time information of each teaching courseware text information, aligning the multiple teaching key text information and the multiple teaching courseware text information on the main timeline to obtain multiple synchronous text information and synchronous associated time information of each synchronous text information; The plurality of synchronous text information and the synchronous associated time information of each synchronous text information are corrected to obtain a plurality of teaching text information and fourth associated time information corresponding to each teaching text information.

3. The method according to claim 2, characterized in that Each of the plurality of synchronous text messages includes a teaching key text message and teaching courseware text information synchronized with the teaching key text message; Correspondingly, the correction of the plurality of synchronous text information and the synchronous associated time information of each synchronous text information to obtain the plurality of teaching text information and the fourth associated time information corresponding to each teaching text information includes: According to the time sequence of the synchronization-related time information, the plurality of synchronization text information are traversed in sequence, and for the traversed synchronization text information, the following steps are performed: Acquire key teaching text information and teaching courseware text information in the synchronous text information; Calculating a first correlation between the teaching key text information and the teaching courseware text information; If it is determined that the first correlation does not reach a preset first threshold, obtaining the start time of the synchronization association time information of the synchronization text information; Calculating a second correlation between the teaching content text information and the teaching courseware text information within a preset time period before and after the start time; If it is determined that the second correlation reaches a preset second threshold, correcting the key teaching text information in the synchronous text information to be the teaching content text information or teaching courseware text information within a preset time period before and after the start time; Obtaining teaching text information according to the corrected synchronous text information; The synchronization associated time information is determined to be the fourth associated time information of the teaching text information.

4. The method according to claim 1, characterized in that: The returning a corresponding reply result to the user terminal according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information includes: Inputting the search content, the target teaching text information and the fourth correlation time information corresponding to the target teaching text information into a preset macro model to determine whether the third correlation between the target teaching text information and the search content reaches a preset third threshold; If it is determined that the third correlation reaches the preset third threshold, the preset large model is used to generate a corresponding reply result based on the question information, the target teaching text information and the fourth associated time information corresponding to the target teaching text information.

5. The method according to claim 4, characterized in that After determining whether the third correlation between the target teaching text information and the search content reaches a preset third threshold, the method further includes: If it is determined that the third correlation does not reach the preset third threshold, the preset large model is used to generate a corresponding answer result according to the question information.

6. The method according to any one of claims 1 to 5, characterized in that: After receiving the question information sent by the user terminal, the method further includes: Input the question information into a preset macro model to determine whether the question information is a content query or a video search; If it is determined that the question information is a content query or a video search, the step of "obtaining corresponding target teaching text information and fourth associated time information corresponding to the target teaching text information from the search engine according to the search content" is executed.

7. A video question-answering device, characterized in that: Applied to service equipment, including: An acquisition module, used to acquire a preset number of teaching videos; The extraction module is used to perform audio extraction processing on each teaching video to obtain teaching audio; An identification module is used to transcribe the teaching audio to obtain a plurality of teaching content text information and first associated time information of each teaching content text information; An input module, used for inputting the plurality of teaching content text information and the first associated time information of each teaching content text information into a preset large language model, so as to output a plurality of teaching key text information and the second associated time information of each teaching key text information; The extraction module is further used to perform frame extraction processing on each teaching video according to a preset frame extraction frequency to obtain multiple teaching pictures; The recognition module is further used to perform text recognition on the multiple teaching pictures to obtain multiple teaching courseware text information and third associated time information of each teaching courseware text information; A fusion module, used for performing information fusion processing on the plurality of teaching key text information and the second associated time information of each teaching key text information and the plurality of teaching courseware text information and the third associated time information of each teaching courseware text information, so as to obtain the fourth associated time information corresponding to the plurality of teaching text information and each teaching text information; A storage module, used for storing the multiple teaching text information corresponding to each teaching video and the fourth associated time information of each teaching text information into the search engine; A receiving module, used for receiving question information sent by a user terminal, wherein the question information includes search content; The acquisition module is further used to acquire corresponding target teaching text information and fourth associated time information corresponding to the target teaching text information from the search engine according to the search content; The sending module is used to return the corresponding reply result to the user terminal according to the target teaching text information and the fourth associated time information corresponding to the target teaching text information.

8. A service device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the video question-answering method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the video question-answering method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images