Question and answer method and device based on multi-fragment retrieval, equipment and medium

Abstract

The invention relates to the technical field of intelligent question answering, financial science and technology and medical health, and discloses a question answering method and device based on multi-fragment retrieval, equipment and a medium, and the method comprises the steps: obtaining demand information and a target video associated with the demand information; inputting the demand information and the target video into a preset fragment retrieval model to enable the preset fragment retrieval model to extract a plurality of time fragments associated with the demand information from the target video; performing multi-modal content extraction on the target video based on each time slice to obtain a multi-modal vector matched with the time slice; and inputting the multi-modal vector into a preset multi-modal large model to enable the preset multi-modal large model to generate reply information for answering the demand information. Not only is question and answer efficiency improved, but also multi-modal retrieval is supported.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent question answering, financial technology, and medical health technology, and in particular to a question answering method, device, equipment, and medium based on multi-segment retrieval. Background Art

[0002] Video RAG (Retrieval-Augmented Generation) is an intelligent framework that integrates video retrieval technology with large generative models. Its core goal is to enable large language models to enhance the accuracy, richness, and interpretability of responses to video-related questions by retrieving key segments from videos. Consequently, it is widely used in technical fields such as fintech, healthcare, and insurance. For example, Video RAG can be used to locate violations in transaction videos, key steps in surgical videos, or critical nodes in car accident videos.

[0003] The inventors realized that the above retrieval method extracts key frames frame by frame, resulting in slow processing speed and low question-answering efficiency. At the same time, it only supports retrieval of video-related modalities, but not other modal retrievals, and has certain limitations. Summary of the Invention

[0004] The present invention provides a question-answering method, apparatus, computer equipment, and medium based on multi-fragment retrieval to solve the technical problems of low question-answering efficiency and lack of support for multimodal retrieval in existing retrieval methods.

[0005] In a first aspect, a question answering method based on multi-segment retrieval is provided, comprising:

[0006] Acquiring demand information and a target video associated with the demand information;

[0007] Inputting the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video;

[0008] Performing multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment;

[0009] The multimodal vector is input into a preset multimodal large model so that the preset multimodal large model generates response information that answers the demand information.

[0010] In a second aspect, a question-answering device based on multi-segment retrieval is provided, comprising:

[0011] A first acquisition module is used to acquire demand information and a target video associated with the demand information;

[0012] A first input module is configured to input the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video;

[0013] A first extraction module, configured to perform multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment;

[0014] The second input module is used to input the multimodal vector into a preset multimodal large model so that the preset multimodal large model generates response information to answer the demand information.

[0015] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned question-answering method based on multi-fragment retrieval are implemented.

[0016] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned question-answering method based on multi-fragment retrieval are implemented.

[0017] In the scheme implemented by the above-mentioned question-answering method, device, computer equipment and storage medium based on multi-segment retrieval, demand information and a target video associated with the demand information are obtained; the demand information and the target video are input into a preset segment retrieval model so that the preset segment retrieval model extracts multiple time segments associated with the demand information from the target video; multimodal content extraction is performed on the target video based on each time segment to obtain a multimodal vector matching the time segment; the multimodal vector is input into a preset multimodal large model so that the preset multimodal large model generates reply information that answers the demand information. In the present invention, the demand information and the target video can be input into the preset segment retrieval model, and multiple time segments can be obtained through the preset segment retrieval model to improve the retrieval efficiency. At the same time, multimodal content extraction can be performed on the target video based on the time segment to obtain a multimodal vector, and then the multimodal vector is input into the preset multimodal large model to generate reply information. While improving the question-answering efficiency, it also supports multimodal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 1 is a flow chart of a question-answering method based on multi-segment retrieval in one embodiment of the present invention;

[0020] Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S30;

[0021] Figure 3 1 is a schematic diagram of a first sub-process of a question-answering method based on multi-segment retrieval in one embodiment of the present invention;

[0022] Figure 4 2 is a schematic diagram of a second sub-process of a question-answering method based on multi-segment retrieval in one embodiment of the present invention;

[0023] Figure 5 3 is a schematic diagram of a third sub-process of a question-answering method based on multi-segment retrieval in one embodiment of the present invention;

[0024] Figure 6 4 is a schematic diagram of a fourth sub-process of a question-answering method based on multi-segment retrieval in one embodiment of the present invention;

[0025] Figure 7 1 is a schematic structural diagram of a question-answering device based on multi-segment retrieval in one embodiment of the present invention;

[0026] Figure 8 is a structural diagram of a computer device in one embodiment of the present invention;

[0027] Figure 9 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0029] The question-answering method based on multi-segment retrieval provided by an embodiment of the present invention can be applied to computer devices, which can be, but are not limited to, various personal computers, laptops, and smart phones. Among them, demand information and a target video associated with the demand information are obtained; the demand information and the target video are input into a preset segment retrieval model so that the preset segment retrieval model extracts multiple time segments associated with the demand information from the target video; multimodal content extraction is performed on the target video based on each of the time segments to obtain a multimodal vector that matches the time segment; the multimodal vector is input into a preset multimodal large model so that the preset multimodal large model generates reply information that answers the demand information, which can improve the efficiency of question-answering while realizing multimodal retrieval. The present invention is described in detail below through specific embodiments.

[0030] See also Figure 1 As shown, Figure 1 A flowchart of a question-answering method based on multi-segment retrieval provided in an embodiment of the present invention includes the following steps:

[0031] S10: Obtain demand information and a target video associated with the demand information.

[0032] The question-answering method based on multi-segment retrieval provided by the present invention can be applied to various technical fields, such as financial technology, medical health, insurance, and daily life. In the field of financial technology, many operations require compliance, so compliance training courses are usually set up. In the compliance training video, the risk case segment in the compliance training video can be quickly located by the question-answering method based on multi-segment retrieval, thereby improving the training efficiency. In the field of medical health, the key operations in the surgical video can be quickly located by the question-answering method based on multi-segment retrieval. For example, in the "Laparoscopic Cholecystectomy" video, the bile duct exposure operation step can be quickly located by the question-answering method based on multi-segment retrieval.

[0033] Users can enter their needs based on their requirements. This information can be text or voice, and voice messages can be converted to text. The needs of users with different identities often vary significantly. For example, a doctor's needs may include the steps of a specific surgery, such as "Bile duct exposure steps for laparoscopic cholecystectomy." Financial professionals' needs may include whether a specific operation is compliant, such as "Key points for double-recording compliance for financial product sales."

[0034] Target videos are videos associated with the requested information, and there can be one or more of them. Target videos are highly correlated with the requested information and are typically selected by the user. For example, if the user's requested information is the procedure for a specific surgery, the target video is typically a surgical case study. If the user's requested information is whether a specific behavior is compliant, the target video is typically a compliance training video. In addition to fintech and healthcare, this technology can also be applied in daily life. For example, users can locate peak hours in surveillance videos by entering "rush hour traffic in a shopping mall." Alternatively, users can use it in traffic safety, such as by entering "cars running red lights" to locate cars that violate regulations in surveillance videos.

[0035] S20: Inputting the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video.

[0036] After obtaining the demand information and the target video, the demand information is input into a preset segment retrieval model. The preset segment retrieval model extracts multiple time segments associated with the demand information from the target video according to the demand information, and then outputs the extracted time segments.

[0037] The preset segment retrieval model can be the moment-DETR (Moment Detection Transformer) model, which can locate multiple relevant time segments in the target video based on text queries. In other words, it can locate multiple time segments in the target video that are relevant to the required information based on the required information. moment-DETR includes a TransformerEncoder, a TransformerDecoder, a Feedforward Neural Network (FFN), and a loss function. The TransformerEncoder encodes the input text into semantic vectors, capturing key information within the text (e.g., "car insurance claim" and "video evidence"). The TransformerDecoder generates "MomentQueries" (moment query vectors) for matching time segments in the video. Each query corresponds to the starting point of the segment to be retrieved. The Feedforward Neural Network (FFN) associates the semantic vectors with timestamps and outputs predicted time intervals. L1 loss (optimizing timestamp prediction accuracy) and IoU loss (optimizing the overlap between predicted and actual segments) are used to ensure accurate localization. In use, the input text (e.g., "How to shoot a car accident video") is first converted into semantic features. Based on the semantic features, multiple MomentQueries are then generated, each corresponding to a possible segment start time. Timestamp prediction: MomentQueries are mapped to specific time intervals (such as the start seconds) and optimized using the L1+IoU loss to maximize the overlap between the predicted segments and the actual relevant segments. Finally, the N most relevant video segment timestamps are returned, where N is an empirical value.

[0038] For example, the demand information is "laparoscopic cholecystectomy bile duct exposure steps", and the surgical case video is obtained at the same time. The demand information and the surgical case video are input into the preset segment retrieval model. The preset segment retrieval model locates the part of the surgical case video about "laparoscopic cholecystectomy bile duct exposure steps" according to the demand information, such as "bile duct anatomy", "gallbladder triangle separation", and "vascular clamp application", and then confirms the timestamps of these three parts to obtain two time segments, which are time segment 1 and time segment 2. Time segment 1: 23:15-25:40 (bile duct exposure operation demonstration, including bile duct anatomy and gallbladder triangle separation), time segment 2: 32:05-33:30 (vascular clamp usage specifications).

[0039] If the demand information is "prohibited language in financial product sales records", the target video can be a compliance training video. The demand information and the compliance training video are input into the preset segment retrieval model. The preset segment retrieval model quickly locates the violation case in the target video based on the demand information, such as "illegal language in the sales promise of 'guaranteed principal and interest'", and confirms the timestamp corresponding to the segment, thereby obtaining the time segment 1: 10:22-11:05 (illegal language in the sales promise of 'guaranteed principal and interest').

[0040] S30: Perform multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment.

[0041] The video corresponding to the time segment is the portion of the target video that matches the required information. After the time segment is determined, matching video segments can be identified in the target video based on the time segment. Multimodal content extraction is then performed to obtain a multimodal vector. Multimodal content extraction can extract keyframes, keyframe summaries, and the audio corresponding to the keyframes from the target video.

[0042] In some embodiments, wherein Figure 2 As shown, a method for generating a multimodal vector is provided. In step S30, multimodal content extraction is performed on the target video based on each time segment to obtain a multimodal vector matching the time segment, specifically comprising the following steps:

[0043] S31: intercepting a corresponding segment from the target video according to the time segment to obtain an original segment;

[0044] S32: performing visual feature extraction on the original segment to obtain image features and text summary respectively;

[0045] S33: extracting audio features from the original segment and converting the extracted audio features into text to obtain a text description;

[0046] S34: Concatenate the image features, the text summary, and the text description into the multimodal vector.

[0047] According to the time segments output by the preset segment detection model, the corresponding segments are cut out in the target video to obtain multiple original segments. The resolutions of the multiple original segments can be unified to the same resolution, such as 224×224. Then the key frames are extracted from the original segments, such as taking 3 frames (start, middle, and end) for each segment, and the image features are extracted through the visual encoder. The pre-trained model is used to input the segment video frame sequence to generate a text summary (such as "multi-angle shooting demonstration of the accident scene"). The segment audio is converted into text to obtain a text description, and keywords are extracted (such as "shooting time" and "road signs"). Finally, the image features, text summaries, and text descriptions are spliced ​​into a multimodal vector. If the image feature is V k , the text summary is desc k , the text is described as asr k , then the multimodal vector F k =[V k ,desc k ,asr k ].

[0048] For example, the requirement information is "Laparoscopic cholecystectomy bile duct exposure steps", and time segment 1 is 23:15-25:40 (bile duct exposure operation demonstration, including bile duct anatomy and gallbladder triangle separation), time segment 2 is 32:05-33:30 (vascular clamp usage specifications), time segment 1 and time segment 2 obtain videos in the target video that match the timestamps of time segment 1 and time segment 2 to obtain original segments 1 and original segments 2, visual feature extraction is performed on original segment 1 to obtain "Use a suction device to push away the fat tissue and expose the common bile duct", audio feature extraction is performed to obtain "Be careful to avoid damaging the portal vein branches", and visual feature extraction is performed on original segment 2 to obtain "The vascular clamp should be clamped at a 45° angle to the bile duct".

[0049] If the demand information is "Prohibited language in double recording of financial product sales", time segment 1 is 10:22-11:05 (the illegal language of the sales promise of 'guaranteed principal and interest'), based on time segment 1, original segment 1 is intercepted from the target video, and visual features are extracted from original segment 1 to obtain "The screen displays 'This product guarantees returns'", and audio features are extracted from original segment 1 to obtain "Don't worry, this product will definitely not lose money."

[0050] S40: Inputting the multimodal vector into a preset multimodal macro model so that the preset multimodal macro model generates response information that answers the demand information.

[0051] After obtaining the multimodal vector, the multimodal vector is input into a preset multimodal large model, which responds to the user's needs based on the multimodal vector to generate a response message. The preset multimodal large model can be a general large language model, or a dedicated large language model associated with the user's question. It is understandable that the demand information can be used as a question input into the preset multimodal large model, or a question associated with the demand information can be input separately. For example, if the demand information is "Laparoscopic cholecystectomy bile duct exposure steps", the question can be "What are the key operational points for bile duct exposure?"

[0052] For example, the multimodal vector F k =[V k ,desc k ,asr k ], in the scenario of “Laparoscopic cholecystectomy bile duct exposure steps”, V k For original segment 1 (23:15-25:40) and original segment 2 (32:05-33:30), desc k To "avoid damaging the portal vein branches" and "the vascular clamp should be clamped at a 45° angle to the bile duct", asr k To avoid damaging the portal vein branches, F k =[V k ,desc k ,asr k Inputting data into the pre-set multimodal large model yields the following: During laparoscopic cholecystectomy, bile duct exposure should be guided by the following: ① Use a suction device to gently push aside the fatty tissue of the hepatoduodenal ligament to expose the common bile duct; ② Maintain a clear field of view during exposure to avoid excessive traction that could damage the portal vein; ③ The vascular clamp should be clamped at a 45° angle to the bile duct to ensure complete closure without affecting surrounding vessels. For detailed instructions, refer to the video clips 23:15-25:40 and 32:05-33:30.

[0053] In the scenario of "Prohibited Words in Double Recording of Financial Product Sales", V k Original segment 1 (10:22-11:05), desc k For "The screen shows 'This product guarantees returns'", asr k To convey the message "Don't worry, this product will definitely not lose money," the preset multimodal large model output is: prohibited language in the dual recording of wealth management product sales includes: the promise of 'guaranteed principal and interest' (such as the statement 'definitely will not lose money' in 10:22-11:05 of the video). According to regulatory requirements, misleading promises must not be used in the sales process.

[0054] In some embodiments, as Figure 3 As shown, the question answering method based on multi-segment retrieval also includes the following steps:

[0055] S51: Generate rule information according to the demand information, wherein the rule information is used to filter original fragments;

[0056] S52: confirming whether the original segment meets the requirements of the demand information according to the rule information;

[0057] S53: If the original segment meets the requirements of the rule information, then entering the step of inputting the multimodal vector into a preset multimodal macro model so that the preset multimodal macro model generates response information to answer the requirement information;

[0058] S54: If the original segment does not meet the requirements of the rule information, the original segment is deleted.

[0059] In addition to including the user's requirements for the video content itself, the demand information can also include requirements for time, location, etc. When the demand information contains the above content, rule information can be generated based on the demand information. The rule information is a constraint on the original clip, which can specifically include time constraints, location constraints, type presets, etc. Specifically, the demand information (such as "laparoscopic surgery bile duct exposure steps") is semantically encoded, and keywords ("bile duct exposure" "surgical steps"), time constraints ("training videos in the past year"), spatial constraints ("laparoscopic perspective"), etc. are extracted. The parsing results are mapped to predefined rule templates to generate structured rule information. Combined with historical interaction data (such as user feedback on past answers), the rule weights are adjusted through reinforcement learning. For example, if the user repeatedly ignores the result of "clip length > 60 seconds", the upper limit of the length will be automatically reduced to 90 seconds.

[0060] Extract the text description of the original clip (such as ASR speech-to-text, keyframe OCR), and calculate the word vector similarity with the rule keywords (threshold ≥ 0.7). Compare the clip timestamp with the rule time interval, with an error of ±1 day (configurable). Check whether the actual length of the clip is within the rule threshold. Use a pre-trained video classification model to determine whether the clip belongs to the specified type of rule (such as the classification confidence of "surgical operation" ≥ 0.85). Assign weights to the matching results of each dimension (such as 40% for keyword matching and 30% for time matching). A total score of ≥ 80 points is considered to be in compliance with the rule.

[0061] For example, if the requirement is "Query the key steps for bile duct exposure in laparoscopic cholecystectomy since 2024, with a duration of 15-90 seconds," the time constraint is after 2024, the time type is "cholecystectomy procedure," the keyword matches are "laparoscopy," "gallbladder," "bile duct," and "exposure," and the clip duration threshold is 15-90 seconds. If an original clip has a timestamp of 2024-03-15 and a duration of 2 minutes and 10 seconds (longer than 90 seconds, not meeting the duration requirement), and a score of 75 (20 points deducted for duration), it is considered non-compliant. Another clip, with a duration of 30 seconds, a keyword similarity of 0.85, and an event type confidence of 0.91, has a score of 0.4 × 0.85 + 0.3 × 1.0 + 0.3 × 0.91 = 0.885, ≥ 80, and is considered compliant.

[0062] In some embodiments, as Figure 4 As shown, the question answering method based on multi-segment retrieval also includes the following steps:

[0063] S60: confirming a rule knowledge base matching the demand information according to the demand information;

[0064] S61: Training the preset segment retrieval model based on the rule knowledge base to optimize the preset segment retrieval model.

[0065] Different technical fields have different rule knowledge bases. For example, in the field of traffic safety, the rule knowledge base can be traffic regulations. The preset segment retrieval model can be trained with traffic safety regulations, allowing the preset segment retrieval model to identify the time segments corresponding to violations such as "running a red light" and "failing to give way."

[0066] In some embodiments, as Figure 5 As shown, the question answering method based on multi-segment retrieval also includes the following steps:

[0067] S70: Acquire a plurality of the time segments, and identify the content type of the video segment corresponding to each of the time segments;

[0068] S71: Based on the content type, a multimodal macro model matching the content type is selected to obtain the preset multimodal macro model.

[0069] A dynamic routing mechanism can be built to select different multimodal big models based on different content types to improve answer accuracy. For example, a financial report video clip (11:30-12:05) includes a CEO explaining a pie chart of the "2024 Insurance Business Revenue Structure." The content type is Data Presentation - Chart Presentation. A professional chart parsing model can be used as the default multimodal big model.

[0070] In some embodiments, as Figure 6 As shown, the question answering method based on multi-segment retrieval also includes the following steps:

[0071] S80: Acquire training data, wherein the training data includes a plurality of training segments, and the plurality of training segments include cause segments and result segments;

[0072] S81: Training the preset segment retrieval model using the training data so that the preset segment retrieval model outputs time segments with causal relationships.

[0073] A causal relationship prediction branch is added to the moment-DETR output, using a graph neural network (GNN) to model inter-segment dependencies. Training data: Labeled "cause segment - result segment" pairs (e.g., "operation error segment → failure result segment") generate causal labeled segment sequences during inference (e.g., "[Cause] Video 05:10-05:15 Operation violation → [Result] Video 12:30-12:35 Equipment failure").

[0074] For example, the demand information is "Reason for a production line shutdown," and the causal chain segments are: 05:10-05:15 (Workers failed to lubricate bearings according to regulations, cause), 12:30-12:35 (Bearing overheating and shutdown, consequence). The preset multimodal large model output is: The production line shutdown was caused by the failure to lubricate the bearings between 05:10-05:15, which led to bearing overheating and locking between 12:30-12:35. It is recommended to check the lubrication system and replenish lubricant.

[0075] It can be seen that in the above scheme, the demand information and the target video can be input into the preset segment retrieval model, and multiple time segments can be obtained through the preset segment retrieval model to improve the retrieval efficiency. At the same time, multimodal content extraction can be performed on the target video based on the time segments to obtain a multimodal vector, and then the multimodal vector is input into the preset multimodal large model to generate reply information. While improving the question and answer efficiency, it also supports multimodal retrieval.

[0076] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] In one embodiment, a question-answering device based on multi-segment retrieval is provided. The question-answering device based on multi-segment retrieval corresponds one-to-one to the question-answering method based on multi-segment retrieval in the above embodiment. Figure 7 As shown, the question-answering device based on multi-segment retrieval comprises a first acquisition module 101, a first input module 102, a first extraction module 103 and a second input module 104. The functional modules are described in detail as follows:

[0078] A first acquisition module 101 is configured to acquire demand information and a target video associated with the demand information;

[0079] A first input module 102 is configured to input the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video;

[0080] A first extraction module 103 is configured to perform multimodal content extraction on the target video based on each time segment to obtain a multimodal vector matching the time segment;

[0081] The second input module 104 is configured to input the multimodal vector into a preset multimodal large model so that the preset multimodal large model generates response information for answering the demand information.

[0082] In one embodiment, the first extraction module 103 is specifically configured to:

[0083] Cutting out corresponding segments from the target video according to the time segments to obtain original segments;

[0084] performing visual feature extraction on the original segment to obtain image features and text summaries respectively;

[0085] Extracting audio features from the original segment and converting the extracted audio features into text to obtain a text description;

[0086] The image features, the text summary, and the text description are concatenated into the multimodal vector.

[0087] In one embodiment, the question-answering device based on multi-segment retrieval further includes a first generation module, a first confirmation module, a jump module, and a deletion module.

[0088] A first generating module is configured to generate rule information according to the requirement information, wherein the rule information is used to filter original segments;

[0089] A first confirmation module is configured to confirm whether the original segment meets the requirements of the demand information according to the rule information;

[0090] a jump module, configured to, if the original segment meets the requirements of the rule information, enter a step of inputting the multimodal vector into a preset multimodal macro model so that the preset multimodal macro model generates response information that answers the requirement information;

[0091] The deleting module is configured to delete the original segment if the original segment does not meet the requirements of the rule information.

[0092] In one embodiment, the question-answering device based on multi-segment retrieval further includes a second confirmation module and a first training module.

[0093] A second confirmation module is used to confirm the rule knowledge base matching the demand information according to the demand information;

[0094] The first training module is used to train the preset segment retrieval model based on the rule knowledge base to optimize the preset segment retrieval model.

[0095] In one embodiment, the question-answering device based on multi-segment retrieval further includes a second acquisition module and a first matching module.

[0096] A second acquisition module is used to acquire a plurality of the time segments and identify the content type of the video segment corresponding to each of the time segments;

[0097] A first matching module is configured to select a multimodal macro model that matches the content type based on the content type to obtain the preset multimodal macro model.

[0098] In one embodiment, the question-answering device based on multi-segment retrieval further includes a third acquisition module and a second training module.

[0099] a third acquisition module, configured to acquire training data, wherein the training data includes a plurality of training segments, and the plurality of training segments include cause segments and result segments;

[0100] The second training module is used to train the preset segment retrieval model using the training data so that the preset segment retrieval model outputs time segments with causal relationships.

[0101] The present invention provides a question-answering device based on multi-segment retrieval, which can input demand information and a target video into a preset segment retrieval model, obtain multiple time segments through the preset segment retrieval model, and improve retrieval efficiency. At the same time, multimodal content extraction can be performed on the target video based on the time segments to obtain a multimodal vector, and then the multimodal vector is input into a preset multimodal large model to generate reply information. While improving the question-answering efficiency, it also supports multimodal retrieval.

[0102] For the specific definition of the question-answering device based on multi-segment retrieval, please refer to the definition of the question-answering method based on multi-segment retrieval above, which will not be repeated here. The various modules in the above-mentioned question-answering device based on multi-segment retrieval can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a question-answering method based on multi-segment retrieval.

[0104] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a question-answering method based on multi-segment retrieval

[0105] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0106] Acquiring demand information and a target video associated with the demand information;

[0107] Inputting the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video;

[0108] Performing multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment;

[0109] The multimodal vector is input into a preset multimodal large model so that the preset multimodal large model generates response information that answers the demand information.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0111] Acquiring demand information and a target video associated with the demand information;

[0112] Inputting the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video;

[0113] Performing multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment;

[0114] The multimodal vector is input into a preset multimodal large model so that the preset multimodal large model generates response information that answers the demand information.

[0115] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0116] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0117] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0118] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A question answering method based on multi-segment retrieval, characterized in that: include: Acquiring demand information and a target video associated with the demand information; Inputting the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video; Performing multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment; The multimodal vector is input into a preset multimodal large model so that the preset multimodal large model generates response information that answers the demand information.

2. The method according to claim 1, wherein The performing multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment includes: Cutting out corresponding segments from the target video according to the time segments to obtain original segments; performing visual feature extraction on the original segment to obtain image features and text summaries respectively; Extracting audio features from the original segment and converting the extracted audio features into text to obtain a text description; The image features, the text summary, and the text description are concatenated into the multimodal vector.

3. The method according to claim 1, wherein The method comprises: generating rule information according to the demand information, wherein the rule information is used to filter original fragments; confirming whether the original segment meets the requirements of the demand information according to the rule information; If the original segment meets the requirements of the rule information, the process proceeds to a step of inputting the multimodal vector into a preset multimodal macro model so that the preset multimodal macro model generates response information that answers the requirement information.

4. The method according to claim 3, wherein After confirming whether the original segment meets the requirements of the demand information according to the rule information, the method includes: If the original segment does not meet the requirement of the rule information, the original segment is deleted.

5. The method according to claim 1, wherein The method comprises: confirming a rule knowledge base matching the demand information according to the demand information; The preset segment retrieval model is trained based on the rule knowledge base to optimize the preset segment retrieval model.

6. The method according to claim 1, wherein The method comprises: Acquire a plurality of the time segments, and identify the content type of the video segment corresponding to each of the time segments; A multimodal big model matching the content type is selected based on the content type to obtain the preset multimodal big model.

7. The method according to claim 1, wherein The method comprises: Acquire training data, wherein the training data includes a plurality of training segments, and the plurality of training segments include cause segments and result segments; The preset segment retrieval model is trained using the training data so that the preset segment retrieval model outputs time segments with causal relationships.

8. A question-answering device based on multi-segment retrieval, characterized in that: include: A first acquisition module is used to acquire demand information and a target video associated with the demand information; A first input module is configured to input the demand information and the target video into a preset segment retrieval model so that the preset segment retrieval model extracts a plurality of time segments associated with the demand information from the target video; A first extraction module, configured to perform multimodal content extraction on the target video based on each of the time segments to obtain a multimodal vector matching the time segment; The second input module is used to input the multimodal vector into a preset multimodal large model so that the preset multimodal large model generates response information to answer the demand information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the question-answering method based on multi-fragment retrieval are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the question-answering method based on multi-segment retrieval are implemented as described in any one of claims 1 to 7.

Images