Question response method based on data enhancement processing and related equipment thereof

Abstract

The invention belongs to the technical field of artificial intelligence, is applied to financial or medical question response scenes, and relates to a question response method based on data enhancement processing and related equipment thereof. Then, when final response data is generated, the original video frame corresponding to the target video is integrated into the question again, finally, question expansion data is generated under integration of the original video frame, the retrieval auxiliary text and the question, and finally, final response data for the target video is generated by utilizing the question expansion data. Therefore, the accuracy of question answering is ensured, and the problem that the content at the beginning or the end of the video is easily ignored due to different time lengths in the process of playing the video content due to the limitation of a video window when question answering is carried out on the video content in the past is avoided. And the problem that the video content cannot be comprehensively understood and the problem response is not accurate enough is solved.

Description

Question Response Methods Based on Data Augmentation Processing and Related Equipment Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to financial or medical problem-solving scenarios, involving problem-solving methods and related equipment based on data augmentation processing. Background Technology

[0002] In recent years, with the development of large language models, large-scale video language models capable of simultaneously understanding video footage and human language have also been rapidly developing. However, these models encounter a serious bottleneck when processing long videos, such as lectures or movies lasting several tens of minutes: the limitation of the context window. This means that when the model considers a problem, it needs to place all relevant information, such as video frames and user questions, into this "window." If the video is too long and the information too abundant, the "window" will be filled, causing the model to be unable to process or fully understand the entire video content. This recognition bottleneck is particularly pronounced in fields such as fintech or healthcare applications where data is complex or video content is rich in information.

[0003] Currently, to address the issue of context window limitations, the mainstream approach mainly involves creating a custom long context model that can accommodate more video frames, minimizing video content loss. Therefore, accurately responding to user questions while fully utilizing video content has become a pressing problem. Summary of the Invention

[0004] The purpose of this application is to propose a question-answering method and related equipment based on data augmentation processing, so as to accurately answer user questions while making full use of video content.

[0005] In a first aspect, embodiments of this application provide a question-and-answer method based on data augmentation processing, employing the following technical solution: The question-and-answer method based on data augmentation processing includes: acquiring a target video; performing frame-by-frame processing on the target video to obtain original video frames; acquiring questions raised by the questioning client regarding the target video; analyzing the questions to generate a retrieval task list; generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputting the original video frames, the retrieval auxiliary text, and the questions into a data integration model to obtain integrated question expansion data; and generating final response data for the target video based on the question expansion data and feeding it back to the questioning client.

[0006] Secondly, embodiments of this application also provide a question-and-answer device based on data augmentation processing, employing the following technical solution: The question-and-answer device based on data augmentation processing includes: a target video acquisition module for acquiring a target video; a video frame-segmentation processing module for performing frame-segmentation processing on the target video to obtain original video frames; a question acquisition module for acquiring questions raised by the questioning client regarding the target video; a retrieval task list generation module for analyzing the questions and generating a retrieval task list; a retrieval auxiliary text generation module for generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; a question expansion data integration module for inputting the original video frames, the retrieval auxiliary text, and the questions into a data integration model to obtain integrated question expansion data; and a final response data generation module for generating final response data for the target video based on the question expansion data and feeding it back to the questioning client.

[0007] Thirdly, embodiments of this application also provide a computer device that adopts the following technical solution: a computer device including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the question-and-response method based on data augmentation processing described above.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium, which adopts the following technical solution: a computer-readable storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, the steps of the question response method based on data augmentation processing described above are implemented.

[0009] Compared with the prior art, the embodiments of this application have the following main advantages: The question-and-answer method based on data augmentation processing described in this application obtains a target video; performs frame segmentation processing to obtain original video frames; obtains the questions raised by the questioning client regarding the target video; analyzes the questions to generate a retrieval task list; generates retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputs the original video frames, retrieval auxiliary text, and questions into a data integration model to obtain integrated question expansion data; and generates final response data for the target video based on the question expansion data and feeds it back to the questioning client. This question-and-answer method not only performs data augmentation on the question by adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question during the final response data generation. Ultimately, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, final response data for the target video is generated, ensuring the accuracy of the question-and-answer process. This avoids the problem of inaccurate responses caused by the limited video window in previous question-and-answer methods targeting video content. The varying lengths of the video playback during playback could lead to overlooking the beginning or end of the video, resulting in a lack of comprehensive understanding of the video content. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 is an exemplary system architecture diagram to which this application can be applied; Figure 2 is a flowchart of an embodiment of the question-and-response method based on data augmentation processing according to this application; Figure 3 is a flowchart of a specific embodiment of step 204 shown in Figure 2; Figure 4 is a flowchart of a specific embodiment of step 205 shown in Figure 2; Figure 5 is a flowchart of a specific embodiment of step 402 shown in Figure 4; Figure 6 is a flowchart of a specific embodiment of step 403 shown in Figure 4; Figure 7 is a flowchart of a specific embodiment of step 406 shown in Figure 4; Figure 8 is a structural schematic diagram of an embodiment of the question-and-response device based on data augmentation processing according to this application; Figure 9 is a structural schematic diagram of an embodiment of the computer device according to this application. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0015] As shown in Figure 1, the system architecture 100 may include a terminal device 101, a network 102, and a server 103. The terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is used as a medium to provide a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0016] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0017] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0018] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0019] It should be noted that the question-and-response method based on data augmentation processing provided in this application embodiment is generally executed by the server, and correspondingly, the question-and-response device based on data augmentation processing is generally located in the server.

[0020] It should be understood that the number of terminal devices, networks, and servers shown in Figure 1 is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0021] Referring again to Figure 2, a flowchart of an embodiment of the question-and-response method based on data augmentation processing according to this application is shown. The question-and-response method based on data augmentation processing includes the following steps: Step 201, acquiring the target video.

[0022] In this embodiment, the target video refers to the video that the questioning end needs to watch. For example, the user at the human-computer interaction interface of the questioning end first watches the target video, and then asks a question based on the target video watched.

[0023] Specifically, the target video includes any type of video containing knowledge information, such as surveillance video or project report video. The specific video content is determined by the question-and-answer task involved in the actual Q&A session. The target video can be obtained by downloading or extracting it from a pre-set video repository.

[0024] Step 202: Perform frame segmentation on the target video to obtain the original video frames.

[0025] In this embodiment, the target video is processed by frame segmentation to obtain original video frames, so as to identify the information content contained in the target video in image frame mode.

[0026] Step 203: Obtain the questions raised by the questioning client regarding the target video.

[0027] In this embodiment, the questioning terminal refers to the viewing terminal of the target video. After the user plays the target video on the viewing terminal, they ask questions based on the information provided in the target video through human-computer interaction and output the questions.

[0028] Specifically, the question can be a text-based question entered by the user on the questioning platform, or a voice question entered by the user on the questioning platform; there is no limitation here.

[0029] Step 204: Analyze the problem and generate a retrieval task list.

[0030] In this embodiment, analyzing the problem refers to parsing and processing the problem, such as parsing out the elements involved in the problem, such as objects, behaviors, and events, and then generating a retrieval task list in the target format. It can also be understood as converting unstructured problem data into structured problem data; for example, if the problem is a text description, converting that text into structured instruction text that is easy for computers to process.

[0031] Step 205: Generate retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames.

[0032] In this embodiment, by parsing the data content in the target video and the original video frames, the information content contained in the target video and the original video frames is obtained. Then, the information content related to the question retrieval is added to the retrieval task list to expand the question and facilitate subsequent response processing.

[0033] Specifically, by generating search auxiliary text to supplement the search task list, the content of both questions and answers is enriched simultaneously, facilitating more accurate and faster question and answer processing in the future.

[0034] Step 206: Input the original video frame, the retrieval auxiliary text, and the question into the data integration model to obtain the integrated question expansion data.

[0035] Specifically, the data integration model includes a large-scale video language model based on LVLM (Large Video-Language Models), capable of simultaneously understanding video footage and human language information. Here, its understanding capabilities are utilized to simultaneously comprehend information from the original video frames, as well as retrieve information from auxiliary text and questions, and perform fusion processing to obtain integrated question-expanded data. Since the question-expanded data is integrated from video footage information and text content information, the accuracy of the question responses is guaranteed.

[0036] Step 207: Based on the question expansion data, generate final response data for the target video and feed it back to the questioning end.

[0037] Specifically, after watching a financial content video on the financial business inquiry platform, the video can be segmented into frames to obtain the original video frames. Then, the questions raised by the financial business inquiry platform regarding the video are obtained, and a search task list is generated. The original video frames and the video content are analyzed to generate search auxiliary text to supplement the search task list. The original video frames, the search auxiliary text, and the questions are then input into a data integration model to obtain integrated question expansion data. Finally, based on the question expansion data, final response data for the video content is generated and fed back to the financial business inquiry platform. This method, applied to financial business question-answering scenarios, enables the expansion of relevant questions, and subsequently, based on the expanded data, generates financial business response content, improving the accuracy of question responses.

[0038] The question-and-answer method based on data augmentation provided in this embodiment not only performs data augmentation on the question by parsing the target video and adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question when generating the final answer data. Finally, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, the final answer data for the target video is generated, ensuring the accuracy of the question-and-answer response. This avoids the problem of inaccurate question-and-answer responses caused by the limitation of the video window, which makes it easy to overlook the beginning or end of the video due to varying video lengths, resulting in an incomplete understanding of the video content.

[0039] In this embodiment, the process involves: acquiring the target video; performing frame segmentation to obtain the original video frames; acquiring the questions raised by the questioning client regarding the target video; analyzing the questions to generate a retrieval task list; generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputting the original video frames, retrieval auxiliary text, and questions into a data integration model to obtain integrated question expansion data; and generating final response data for the target video based on the question expansion data and feeding it back to the questioning client. This question-and-answer method not only performs data augmentation on the question by adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question during the final response data generation. Ultimately, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, final response data for the target video is generated, ensuring the accuracy of the question-and-answer process. This avoids the problem of inaccurate responses caused by the limited video window in previous question-and-answer methods targeting video content. The varying lengths of the video playback during playback could lead to overlooking the beginning or end of the video, resulting in a lack of comprehensive understanding of the video content.

[0040] In this embodiment, the question-and-answer method based on data augmentation processing can be applied to both financial service response processing and healthcare service response processing. It can generate more accurate response results by expanding the scope of question data. Specifically, for example, when a financial customer has questions after reviewing the financial service process, they can ask specific questions. In this case, the data augmentation-based question-and-answer method can automatically integrate the financial service process as relevant knowledge into the question, facilitating accurate response generation.

[0041] In this embodiment, the step of performing frame segmentation processing on the target video to obtain original video frames includes: performing frame segmentation processing on the target video according to a preset frame segmentation time interval to obtain all frame images contained in the target video, which are used as the original video frames.

[0042] Specifically, the target video is divided into continuous images according to a preset frame interval, which facilitates the analysis of the information content contained in the target video in the image dimension.

[0043] Referring again to Figure 3, which is a flowchart of a specific embodiment of step 204 shown in Figure 2, the steps include: Step 301, using syntactic analysis to identify the question parsing template corresponding to the question; specifically, using syntactic analysis to identify the noun objects, verb action data, and event names in the question, and then, based on different question structure methods, identifying the corresponding question parsing template.

[0044] Step 302: Parse the question according to the question parsing template to obtain the parsing result in the target structured format, wherein the target structured format includes JSON format; specifically, the question parsing template reserves a number of fill items for filling the data obtained by parsing, such as: filling name objects, filling event names, etc.

[0045] Step 303: Generate a retrieval task list based on the parsing results of the target structured format.

[0046] By converting the problem from a descriptive text into structured text, it becomes easier for computers to recognize and process it.

[0047] Referring again to Figure 4, which is a flowchart of a specific embodiment of step 205 shown in Figure 2, the steps include: Step 401, inputting the target video into a preset video parsing model; Step 402, using the speech parsing component in the video parsing model to obtain the speech text data contained in the target video; Step 403, using the image content recognition component in the video parsing model to obtain the image text data contained in the video image stream of the target video; Step 404, integrating the speech text data and the image text data as text retrieval resources; Step 405, by performing semantic parsing processing on the text retrieval resources, identifying the text data content contained in the text retrieval resources that conforms to the retrieval task list; Step 406, filtering the original video frames according to the retrieval task list to obtain key video frames; Step 407, performing textual description on the image information contained in all key video frames to obtain textual description results corresponding to each key video frame; Step 408, generating the retrieval auxiliary text using the text data content contained in the text retrieval resources that conforms to the retrieval task list and the textual description results.

[0048] Specifically, this method identifies and analyzes the audio data, text data, and object information contained in the images within the target video, thereby comprehensively understanding the target video and ensuring the accuracy of subsequent question responses. Furthermore, this processing method avoids the problem of incomplete video content recognition caused by excessively small video parsing windows in previous methods.

[0049] Referring again to Figure 5, which is a flowchart of a specific embodiment of step 402 shown in Figure 4, the steps include: Step 501, performing video output format recognition on the target video to determine whether the target video has audio subtitles during output, wherein the video output format specifies whether the target video has audio subtitles during output; Step 502, if the target video has audio subtitles during output, then using a preset crawling tool to crawl the audio subtitles from the target video as the audio text data, wherein the crawling tool includes a Whisper-based text crawling tool; Step 503, if the target video does not have audio subtitles during output, then using speech recognition technology to extract the audio text data from the target video.

[0050] Specifically, when acquiring audio data from a target video, the process begins by determining whether the video contains audio subtitles. Based on whether it contains audio subtitles, different audio-text data extraction methods are selected to acquire the audio-text data, thus improving the flexibility of target video recognition and understanding.

[0051] Referring again to Figure 6, which is a flowchart of a specific embodiment of step 403 shown in Figure 4, the steps include: Step 601, using the OCR recognition tool in the video parsing model to identify the image information of the text style contained in each image of the video image stream in the target video, and the position information of the image information of each text style in the corresponding image, wherein the OCR recognition tool includes the EasyOCR tool; Step 602, based on the position information of the image information of each text style in the corresponding image, performing OCR optical technology extraction to extract the image information of each text style; Step 603, using the preset text comparison library in the OCR recognition component to determine the text content corresponding to the image information of each text style, and by summarizing and organizing, obtaining the image text data contained in the video image stream of the target video.

[0052] Specifically, by utilizing OCR optical technology and text recognition principles, the text data contained in the video frame during playback of the target video is identified, such as text data in the video playback background and text data in the poster in the video. Here, optical recognition is directly used for comparison and recognition, which can significantly improve the efficiency of text recognition.

[0053] Referring again to Figure 7, which is a flowchart of a specific embodiment of step 406 shown in Figure 4, the step includes: Step 701, using the original video frames as image detection resources, and detecting target video frames that match the retrieval task list among all original video frames by performing image target detection on the image detection resources. Specifically, when performing image target detection on the image detection resources, a preset image object detection tool can be used. For example, if the retrieval task list includes retrieving a certain object, the original video frames containing the object are retrieved as target video frames by performing image target detection on the image detection resources.

[0054] Step 702: Mark all target video frames as the key video frames.

[0055] In this embodiment, the step of textualizing the image information contained in all key video frames to obtain textual description results corresponding to each key video frame specifically includes: identifying the entity objects, object behaviors, and behavior results contained in the key video frames; generating the textual description by combining object identifiers, object behaviors, and behavior results; and organizing the textual descriptions to obtain the textual description results contained in each of the key video frames. For example: the current key video frame contains vehicle A and vehicle B, vehicle A rear-ended vehicle B, and vehicle B's rear bumper is broken.

[0056] In this embodiment, the process involves: acquiring the target video; performing frame segmentation to obtain the original video frames; acquiring the questions raised by the questioning client regarding the target video; analyzing the questions to generate a retrieval task list; generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputting the original video frames, retrieval auxiliary text, and questions into a data integration model to obtain integrated question expansion data; and generating final response data for the target video based on the question expansion data and feeding it back to the questioning client. This question-and-answer method not only performs data augmentation on the question by adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question during the final response data generation. Ultimately, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, final response data for the target video is generated, ensuring the accuracy of the question-and-answer process. This avoids the problem of inaccurate responses caused by the limited video window in previous question-and-answer methods targeting video content. The varying lengths of the video playback during playback could lead to overlooking the beginning or end of the video, resulting in a lack of comprehensive understanding of the video content.

[0057] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0058] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0059] In this embodiment, the process involves: acquiring the target video; performing frame segmentation to obtain the original video frames; acquiring the questions raised by the questioning client regarding the target video; analyzing the questions to generate a retrieval task list; generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputting the original video frames, retrieval auxiliary text, and questions into a data integration model to obtain integrated question expansion data; and generating final response data for the target video based on the question expansion data and feeding it back to the questioning client. This question-and-answer method not only performs data augmentation on the question by adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question during the final response data generation. Ultimately, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, final response data for the target video is generated, ensuring the accuracy of the question-and-answer process. This avoids the problem of inaccurate responses caused by the limited video window in previous question-and-answer methods targeting video content. The varying lengths of the video playback during playback could lead to overlooking the beginning or end of the video, resulting in a lack of comprehensive understanding of the video content.

[0060] Referring further to FIG8, as an implementation of the method shown in FIG2 above, this application provides an embodiment of a question response device based on data augmentation processing. This device embodiment corresponds to the method embodiment shown in FIG2, and the device can be specifically applied to various electronic devices.

[0061] As shown in Figure 8, the question response device 800 based on data augmentation processing described in this embodiment includes: a target video acquisition module 801, a video frame processing module 802, a question acquisition module 803, a retrieval task list generation module 804, a retrieval auxiliary text generation module 805, a question extended data integration module 806, and a final response data generation module 807. The system comprises: a target video acquisition module 801 for acquiring a target video; a video frame-segmentation processing module 802 for performing frame-segmentation processing on the target video to obtain original video frames; a question acquisition module 803 for acquiring questions raised by the questioning client regarding the target video; a retrieval task list generation module 804 for analyzing the questions and generating a retrieval task list; a retrieval auxiliary text generation module 805 for generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; a question expansion data integration module 806 for inputting the original video frames, the retrieval auxiliary text, and the questions into a data integration model to obtain integrated question expansion data; and a final response data generation module 807 for generating final response data for the target video based on the question expansion data and feeding it back to the questioning client.

[0062] This application involves: acquiring a target video; performing frame segmentation to obtain original video frames; acquiring questions raised by the questioning client regarding the target video; analyzing the questions to generate a retrieval task list; generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputting the original video frames, retrieval auxiliary text, and questions into a data integration model to obtain integrated question expansion data; and generating final response data for the target video based on the question expansion data and feeding it back to the questioning client. This question-and-answer method not only performs data augmentation on the question by adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question during the final response data generation. Ultimately, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, final response data for the target video is generated, ensuring the accuracy of the question-and-answer process. This avoids the problem of inaccurate responses caused by the limited video window in previous question-and-answer methods targeting video content. The varying lengths of the video playback during playback could lead to overlooking the beginning or end of the video, resulting in a lack of comprehensive understanding of the video content.

[0063] In this embodiment, the retrieval task list generation module 804 includes a question parsing template identification unit, a question parsing execution unit, and a retrieval task list generation unit. Specifically: the question parsing template identification unit is used to identify the question parsing template corresponding to the question using syntactic analysis; the question parsing execution unit is used to parse the question according to the question parsing template to obtain a parsing result in a target structured format, wherein the target structured format includes JSON format; and the retrieval task list generation unit is used to generate a retrieval task list based on the parsing result of the target structured format.

[0064] In this embodiment, the retrieval-aided text generation module 805 includes a target video parsing input unit, a voice text data acquisition unit, an image text data acquisition unit, a text retrieval resource integration unit, a text retrieval resource recognition unit, a key video frame filtering unit, a textual description processing unit, and a retrieval-aided text generation unit. The system includes: a target video parsing input unit for inputting the target video into a preset video parsing model; a speech and text data acquisition unit for acquiring speech and text data contained in the target video using the speech parsing component in the video parsing model; an image and text data acquisition unit for acquiring image and text data contained in the video image stream of the target video using the image content recognition component in the video parsing model; a text retrieval resource integration unit for integrating the speech and text data and the image and text data as text retrieval resources; a text retrieval resource conformance recognition unit for recognizing text data content that conforms to the retrieval task list by performing semantic parsing processing on the text retrieval resources; a key video frame filtering unit for filtering the original video frames according to the retrieval task list to obtain key video frames; a textual description processing unit for performing textual description on the image information contained in all key video frames to obtain textual description results corresponding to each key video frame; and a retrieval auxiliary text generation unit for generating the retrieval auxiliary text using the text data content that conforms to the retrieval task list contained in the text retrieval resources and the textual description results.

[0065] In this embodiment, the voice-text data acquisition unit includes a voice-subtitle judgment subunit, a first branch processing subunit, and a second branch processing subunit. Specifically: the voice-subtitle judgment subunit is used to perform video output format recognition on the target video to determine whether the target video has voice-subtitles when output, wherein the video output format specifies whether the target video has voice-subtitles when output; the first branch processing subunit is used to extract the voice-subtitles from the target video as the voice-text data using a preset extraction tool if the target video has voice-subtitles when output; the second branch processing subunit is used to extract the voice-text data from the target video using speech recognition technology if the target video does not have voice-subtitles when output.

[0066] In this embodiment, the image text data acquisition unit includes an OCR preliminary recognition subunit, an OCR optical extraction subunit, and an OCR recognition determination subunit. Specifically: the OCR preliminary recognition subunit is used to identify the image information of each text style contained in each image of the video image stream in the target video, as well as the position information of each text style image information in the corresponding image, using the OCR recognition tool in the video parsing model; the OCR optical extraction subunit is used to extract the image information of each text style using OCR optical technology based on the position information of each text style image information in the corresponding image; the OCR recognition determination subunit is used to determine the text content corresponding to the image information of each text style using a preset text comparison library in the OCR recognition component, and obtain the image text data contained in the video image stream of the target video by summarizing and organizing the data.

[0067] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0068] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0069] To address the aforementioned technical problems, this application also provides a computer device. Please refer to Figure 9 for details; Figure 9 is a basic structural block diagram of the computer device according to this embodiment.

[0070] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected via a system bus. It should be noted that Figure 9 only shows a computer device 9 with components 9a, 9b, and 9c; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0071] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0072] The memory 9a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 9a may be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 9a may also be an external storage device of the computer device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 9. Of course, the memory 9a may also include both the internal storage unit and its external storage device of the computer device 9. In this embodiment, the memory 9a is typically used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions for question-and-answer methods based on data augmentation processing. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or will be output.

[0073] In some embodiments, the processor 9b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 9b is typically used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to execute computer-readable instructions stored in the memory 9a or to process data, for example, to execute computer-readable instructions of the question-and-answer method based on data augmentation processing.

[0074] The network interface 9c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 9 and other electronic devices.

[0075] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in question-and-answer scenarios. This application acquires a target video; performs frame-by-frame processing to obtain original video frames; acquires the questions raised by the questioning client regarding the target video; analyzes the questions to generate a retrieval task list; generates retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputs the original video frames, retrieval auxiliary text, and questions into a data integration model to obtain integrated question expansion data; and generates final response data for the target video based on the question expansion data and feeds it back to the questioning client. This question-and-answer method not only performs data augmentation on the question by adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question during the final response data generation. Ultimately, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, final response data for the target video is generated, ensuring the accuracy of the question-and-answer process. This avoids the problem of inaccurate responses caused by the limited video window in previous question-and-answer methods targeting video content. The varying lengths of the video playback during playback could lead to overlooking the beginning or end of the video, resulting in a lack of comprehensive understanding of the video content.

[0076] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the question-and-response method based on data augmentation processing as described above.

[0077] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in question-answering scenarios. This application acquires a target video; performs frame segmentation processing to obtain original video frames; acquires the questions raised by the questioner regarding the target video; analyzes the questions to generate a retrieval task list; generates retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; inputs the original video frames, retrieval auxiliary text, and questions into a data integration model to obtain integrated question expansion data; and generates final response data for the target video based on the question expansion data and feeds it back to the questioner. This question-and-answer method not only performs data augmentation on the question by adding retrieval auxiliary text, but also integrates the original video frames corresponding to the target video into the question during the final response data generation. Ultimately, with the integration of the original video frames, retrieval auxiliary text, and the question, extended question data is generated. Finally, using this extended question data, final response data for the target video is generated, ensuring the accuracy of the question-and-answer process. This avoids the problem of inaccurate responses caused by the limited video window in previous question-and-answer methods targeting video content. The varying lengths of the video playback during playback could lead to overlooking the beginning or end of the video, resulting in a lack of comprehensive understanding of the video content.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0079] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

Claims

1. A question-response method based on data augmentation processing, characterized in that, The steps include: acquiring the target video; The target video is segmented into frames to obtain the original video frames; the questions raised by the client regarding the target video are obtained. The problem is analyzed to generate a retrieval task list; based on the target video and the original video frame, retrieval auxiliary text is generated to supplement the retrieval task list; the original video frame, the retrieval auxiliary text, and the problem are input into the data integration model to obtain integrated and processed problem expansion data; Based on the expanded data of the question, the final response data for the target video is generated and fed back to the questioning end.

2. The question-response method based on data augmentation processing according to claim 1, characterized in that, The step of performing frame segmentation processing on the target video to obtain original video frames includes: performing frame segmentation processing on the target video according to a preset frame segmentation time interval to obtain all frame images contained in the target video, which are used as the original video frames.

3. The question-response method based on data augmentation processing according to claim 1, characterized in that, The step of analyzing the question and generating a retrieval task list includes: using syntactic analysis to identify the question parsing template corresponding to the question; parsing the question according to the question parsing template to obtain a parsing result in a target structured format, wherein the target structured format includes JSON format; and generating a retrieval task list based on the parsing result in the target structured format.

4. The question-response method based on data augmentation processing according to claim 1, characterized in that, The step of generating retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames includes: inputting the target video into a preset video parsing model; using the speech parsing component in the video parsing model to obtain the speech text data contained in the target video; using the image content recognition component in the video parsing model to obtain the image text data contained in the video image stream of the target video; integrating the speech text data and the image text data as text retrieval resources; performing semantic parsing processing on the text retrieval resources to identify the text data content contained in the text retrieval resources that conforms to the retrieval task list; filtering the original video frames according to the retrieval task list to obtain key video frames; performing textual description on the image information contained in all key video frames to obtain textual description results corresponding to all key video frames; and generating the retrieval auxiliary text using the text data content contained in the text retrieval resources that conforms to the retrieval task list and the textual description results.

5. The question-response method based on data augmentation processing according to claim 4, characterized in that, The step of using the speech parsing component in the video parsing model to obtain the speech text data contained in the target video includes: performing video output format recognition on the target video to determine whether the target video has speech subtitles when output, wherein the video output format specifies whether the target video has speech subtitles when output; if the target video has speech subtitles when output, then using a preset capture tool to capture the speech subtitles from the target video as the speech text data; if the target video does not have speech subtitles when output, then using speech recognition technology to extract the speech text data from the target video.

6. The question-response method based on data augmentation processing according to claim 4, characterized in that, The step of using the image content recognition component in the video parsing model to obtain the image text data contained in the video image stream of the target video includes: using the OCR recognition tool in the video parsing model to identify the image information of the text style contained in each image of the video image stream of the target video and the position information of the image information of each text style in the corresponding image; performing OCR optical technology extraction based on the position information of the image information of each text style in the corresponding image to extract the image information of each text style; using the preset text comparison library in the OCR recognition component to determine the text content corresponding to the image information of each text style, and obtaining the image text data contained in the video image stream of the target video by summarizing and organizing.

7. The question-response method based on data augmentation processing according to claim 4, characterized in that, The step of filtering the original video frames according to the retrieval task list to obtain key video frames includes: using the original video frames as image detection resources, performing image target detection on the image detection resources, detecting target video frames that match the retrieval task list among all the original video frames, wherein, when performing image target detection on the image detection resources, a preset image object detection tool can be used; and marking all target video frames as the key video frames.

8. A question-response device based on data augmentation processing, characterized in that, include: The target video acquisition module is used to acquire the target video. The video frame segmentation processing module is used to perform frame segmentation processing on the target video to obtain the original video frames; The question acquisition module is used to acquire questions raised by the questioning client regarding the target video; the retrieval task list generation module is used to analyze the questions and generate a retrieval task list. The retrieval auxiliary text generation module is used to generate retrieval auxiliary text to supplement the retrieval task list based on the target video and the original video frames; the question expansion data integration module is used to input the original video frames, the retrieval auxiliary text and the question into the data integration model to obtain integrated question expansion data; the final response data generation module is used to generate final response data for the target video based on the question expansion data and feed it back to the questioning end.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the question-and-response method based on data augmentation processing as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by a processor, implement the steps of the question-response method based on data augmentation processing as described in any one of claims 1 to 7.

Images