Blind assistant video question and answer method based on visual evidence extraction and answerability determination

Abstract

The application discloses a blind person-assisted video question and answer method based on visual evidence extraction and answerability judgment, which comprises the following steps: acquiring video input and user questions and performing pretreatment; guiding a multi-modal large model to extract visual evidence based on a structured prompt word; judging answerability based on the visual evidence; generating a final answer or an unconfirmed statement according to the answerability; and outputting a structured result. Through the chain mechanism of 'evidence first, judgment second and answer last', the application explicitly constrains the multi-modal large model to generate answers based on visible evidence, can significantly reduce the probability of producing hallucinations, and thus improves the safety and reliability of the model when facing complex blind shooting scenes. When there is insufficient evidence, the application does not forcibly output an incorrect answer, but generates an unconfirmed statement and shooting adjustment suggestions, which meet the actual needs in the blind person-assisted scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, multimodal large model, and information accessibility, specifically to an assistive video question-answering method for blind users. This method guides a multimodal large model to first output directly visible visual evidence from the video that is relevant to the question, using structured prompts. It then determines whether the video provides sufficient evidence for answerability, ultimately generating a safe, reliable, and interpretable answer or an unconfirmed statement along with suggestions for adjusting the shooting. This method is applicable to scenarios involving blind assistance, accessible interaction, and visual information acquisition. Background Technology

[0002] With the development of multimodal large models, video question answering technology has been widely used in life assistance, content understanding, and intelligent interaction. However, most existing video question answering systems adopt a direct answer generation method, that is, after inputting a video and a question, the model directly outputs the answer. This type of method can achieve good results in scenarios where visual information is complete and clear, but in actual videos shot by blind people, it often faces the following problems: First, the video footage may have serious blurring, reflections, obstructions, targets outside the frame, or distances that are too far or too close, making it difficult to obtain accurate and complete key visual information. Second, current models often lack the ability to distinguish between "information gaps" and "poor image quality," and are prone to making guessing answers based on linguistic priors when there is insufficient evidence, thus creating illusions or even outputting dangerously erroneous information.

[0003] Third, existing methods often lack explicit logical modeling of the relationship between visual evidence, answerability, and the final answer, resulting in a lack of interpretability in the system output and hindering subsequent training and error analysis.

[0004] Therefore, there is an urgent need for a technical solution that enables the model to identify and extract visual evidence that can be used to answer questions before answering, and then determine whether this evidence is sufficient. Only when the visual evidence is sufficient will a final answer be generated. If the evidence is insufficient, the model will refuse to answer blindly, give an unconfirmed statement, and further provide suggestions for shooting adjustments, thereby improving the safety and reliability of the video question-and-answer system for the blind. Summary of the Invention

[0005] This invention aims to overcome the problem in existing technologies where multimodal large models are prone to hallucinations and fabrication due to missing information or poor image quality in video question-and-answer scenarios for the blind. It provides a method for assistive video question-and-answer for the blind based on visual evidence extraction and answerability determination.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] A Video Question-Answering Method for the Blind Based on Visual Evidence Extraction and Answerability Assessment

[0008] The method includes the following steps: S110: Acquire video input and user questions, and perform preprocessing; receive first-person perspective video taken by a blind user and the natural language questions raised, decode the video and sample frames to obtain a frame sequence, and normalize the size of the sampled frames; S120, Based on structured prompts, guide the multimodal large model to extract visual evidence; input the frame sequence and user question into the multimodal large model, guide the model to identify the type of visual information required to answer the question through structured prompts, and extract directly visible and question-related visual evidence from the video frames, while generating a visual evidence description containing frame-level positioning information; the structured prompts include at least the following constraints: extraction is based only on information directly visible in the video frames, and speculative evidence or common-sense supplementation is prohibited for content that cannot be directly confirmed from the video; S130, Determine answerability based on visual evidence; match the extracted visual evidence with the user's question, calculate the evidence sufficiency score, and determine whether the answerability is "sufficient" or "insufficient" based on the comparison result of the score with a preset threshold; at the same time, the model outputs an answerability confidence score that is positively correlated with the sufficiency of evidence and negatively correlated with video quality issues; the answerability determination is independent of the visual evidence extraction, and even if the visual evidence is insufficient to support the answer, the extracted visual evidence is still output; S140, generate a final answer or an unconfirmed statement based on answerability; if answerability is determined to be "sufficient", generate a natural language answer based on the question and visual evidence; if answerability is determined to be "insufficient", generate a statement that the answer to the current question cannot be confirmed, and generate a shooting adjustment suggestion to obtain clearer visual information. S150, Output Structured Results: Organize visual evidence, answerability determination results, confidence levels, and final answers or statements of nonconfirmation into structured results in JSON or tabular format and output them.

[0009] Furthermore, step S110, which involves acquiring video input and user questions and performing preprocessing, specifically includes: S1101, receiving a first-person perspective video taken by a blind user and the natural language questions they raise; the natural language questions are obtained through text input, voice input transcribed by speech recognition, or synchronous questioning during video recording. S1102, after decoding the video, frame sampling is performed at a fixed frame rate, and the number of sampled frames does not exceed a preset frame count limit; when the number of frames obtained by sampling at the fixed frame rate exceeds the preset frame count limit, the video is uniformly sampled to ensure that the number of frames in the final input model does not exceed the preset frame count limit; the fixed frame rate is preferably 0.5fps to 1fps, and the preset frame count limit is preferably 10 to 20 frames; S1103, normalize the size of the sampled frame: while maintaining the aspect ratio, scale the sampled frame so that the total number of pixels in a single frame image is between a preset lower pixel limit and a preset upper pixel limit; the preset lower pixel limit is preferably 32×32 pixels, and the preset upper pixel limit is preferably 336×336~448×448 pixels.

[0010] Furthermore, the visual evidence extraction described in step S120 specifically includes: S1201, Construct structured prompts, which include the following functional modules: role setting module, answer principle module, perspective and expression constraint module, uncertainty constraint module, annotation requirement module, evidence requirement module, label description module, and output rule module. The specific content of each functional module is described in detail in the specific implementation method. S1202, Identify the type of visual information required for the problem, and determine the type of visual information it depends on based on the problem type. The visual information type includes color, text, objects, image icons, depth distance size, person status, overall environment information, position and direction, and other information. S1203, Identify visible visual evidence directly related to the question from the frame sequence. The model only retains visual evidence that can be directly seen from the video and supports the answer. For content that cannot be directly confirmed from the video, speculative evidence or common sense supplements shall not be generated. S1204, Generate visual evidence description and frame-level positioning information. Provide a brief text description for each piece of visual evidence and mark the corresponding frame number or frame interval. Each piece of visual evidence includes at least a frame number or frame interval field, an evidence type field, an evidence text description field, and a support marker field. The evidence type includes objects, text, spatial location, image quality, people, environment, and others. When the visual evidence is related to the question but is insufficient on its own to establish or confirm the answer, the support marker field is set to no.

[0011] Furthermore, the responsiveness determination in step S130 specifically includes: S1301, Determine whether the visual evidence extracted in step S120 is sufficient: Match each piece of visual evidence with the user's question and the type of visual information required to answer the question, and calculate the sufficiency score of the evidence. , ,in, Indicates the first One piece of visual evidence, This indicates a question raised by the user. This represents the set of visual information types required to answer a question. Indicates the first The degree of support of visual evidence for answering the question. Indicates the corresponding weight. The number of visual evidence items; when the sufficiency of evidence score is... Greater than or equal to the preset threshold When the answerability is sufficient, the sufficiency of evidence is judged as "adequate"; when the sufficiency of evidence score is... Less than the preset threshold When the answerability is insufficient, the answerability is determined as "insufficient"; S1302, In a preferred embodiment, the calculation and threshold comparison process of the evidence sufficiency score is implicitly completed by the multimodal large model under the guidance of structured prompts: a pre-constructed answer existence label space is built, which includes conditions such as the answer being clearly visible, blurring due to motion or shaking, glare, missing information, excessive distance, excessive distance, target occlusion, and others; the model maps the answerability determination result to the answer existence label space, and when the mapping result is "the answer is clearly visible," it is equivalent to... The responsiveness is determined to be "sufficient"; when the mapping result is another label, it is equivalent to The answerability is judged as "insufficient", and the mapped label is output as the specific reason for insufficient evidence; S1303, output answerability confidence score. The model gives an integer score of 0 to 100 for "whether the current video can correctly answer the question", which represents the degree of confidence that the question can be reliably answered based solely on visible evidence. The confidence score is positively correlated with the sufficiency of evidence and negatively correlated with problems such as blurriness, reflection, occlusion, target out of frame, and distance being too far or too close.

[0012] Furthermore, step S140, which involves generating a final answer or a statement that cannot be confirmed, specifically includes: S1401, if the answerability determination in step S130 is "sufficient", then a natural language answer that is understandable and operable for blind users is generated based on the question and visual evidence; the natural language answer is expressed from the user's first-person perspective, with "you" as the subject, and does not contain visual attributes that are not directly related to the question unless the user explicitly asks in the question. S1402, if the answerability determination in step S130 is "insufficient", then based on the question and the missing visual information, a statement that the answer to the current question cannot be confirmed is generated; the statement that cannot be confirmed describes the content that is actually visible in the current video, and explains the specific reasons for not being able to confirm the answer based on the actual situation of the video. The specific reasons include the target being occluded, the target being out of the frame, the target being too small, the image being blurry, reflection or glare, the target being only partially visible or key details not being clearly presented; it is prohibited to output only a general statement that cannot be confirmed without specific reasons; S1403, Based on the specific reason why the answer cannot be confirmed, generate a brief shooting adjustment suggestion, which includes adjusting the lens angle, moving closer to the target, increasing the distance, reducing shaking, improving lighting, or avoiding reflections, in order to re-acquire more complete and clearer visual information.

[0013] Furthermore, the output structured result described in step S150 specifically includes: S1501, when the responsiveness of step S130 is determined to be "sufficient", the visual evidence, the responsiveness determined to be "sufficient", the confidence level and the final answer are organized into a structured output result. S1502, when the responsiveness of step S130 is determined to be "insufficient", the visual evidence, the responsiveness determined to be "insufficient", the confidence level, the specific reasons for the insufficient evidence, the statements that cannot be confirmed, and the shooting adjustment suggestions are organized into a structured output result. S1503, the structured results are saved in JSON or tabular format, including at least a visual evidence list field, an answerability field, a confidence field, and an answer field; wherein, the answerability field takes the value of "sufficient" or "insufficient", and the confidence field takes the value of an integer from 0 to 100; the structured results are used for subsequent data analysis, or as structured annotations during the training phase for fine-tuning the target model.

[0014] The beneficial effects of this invention include: First, this invention uses a chain mechanism of "evidence first, judgment then response" to explicitly constrain multimodal large models to generate responses based on visible evidence, which can significantly reduce the probability of hallucinations and thus improve the safety and reliability of the model when facing complex blind people's shooting scenarios.

[0015] Second, this invention does not force an incorrect answer when there is insufficient evidence, but instead generates statements that cannot be confirmed and suggestions for adjusting the shooting, which meets the actual needs in assistive scenarios for the blind.

[0016] Third, the present invention outputs structured results, which facilitates subsequent training, evaluation, error analysis and data accumulation, and helps to form a continuously iterative data loop.

[0017] Fourth, this invention has strong versatility. It is not only applicable to video-assisted question answering for the blind, but can also be extended to medical image question answering, low-quality video understanding, or trusted multimodal interaction in security-sensitive scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart of a video question-and-answer method for the blind based on visual evidence extraction and responsiveness determination, provided in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. The drawings are simplified schematic diagrams, used only to illustrate the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] Example 1

[0021] like Figure 1 As shown, this embodiment provides a video question-answering method for the blind based on visual evidence extraction and responsiveness determination. This method guides a multimodal large model to first extract directly visible and question-related visual evidence from the video using structured prompts. Then, it determines whether the current video is sufficient to support an answer based on the visual evidence. Finally, based on the determination result, it generates a final answer or a statement of inconclusiveness and suggestions for adjusting the shooting, thereby mitigating the problem of dangerous hallucinations in blind-assisted scenarios and improving safety and reliability.

[0022] Specifically, the method includes the following steps: S110: Obtain video input and user questions, and perform preprocessing.

[0023] Specifically, the system receives first-person perspective videos taken by blind users and the natural language questions they ask. The videos can be taken by users using devices such as mobile phones or chest cameras, and the questions can be obtained through text input, voice input, or simultaneous questioning during video recording.

[0024] Subsequently, the video is decoded and frame sampled. In a preferred embodiment, sampling is performed at a fixed frame rate of 0.5 fps, with an upper limit of 10 frames sampled. When the video is long and the number of frames obtained at 0.5 fps exceeds 10, the entire video is uniformly subtracted to ensure that the final input model does not exceed 10 frames. In another embodiment, the fixed frame rate can be selected within the range of 0.5 fps to 1 fps, or it can be adaptively adjusted according to the task type. Using a sampling method with a fixed frame rate and an upper limit on the number of frames ensures that videos of varying lengths have consistent information density when input to the model, while controlling the input length and inference overhead of the model.

[0025] Furthermore, the sampled frames are normalized in size: while maintaining the aspect ratio, the sampled frames are scaled so that the total number of pixels in a single frame is between a preset lower limit and a preset upper limit. In a preferred embodiment, the preset lower limit is 32×32 pixels, and the preset upper limit is 336×336 pixels; when the total number of pixels in a frame is below the lower limit, it is enlarged, and when it is above the upper limit, it is reduced. This normalization method balances the recognizability of small targets such as text in the image with computational overhead within a limited model context length.

[0026] S120 is a multimodal large model that extracts visual evidence based on structured cue words.

[0027] Specifically, the frame sequence and user question obtained in step S110 are input into the multimodal large model, and the model is guided by structured prompts to output only visual evidence that is directly visible in the video and related to the answer to the question. For content that cannot be directly confirmed from the video, speculative evidence or common sense supplements should not be generated.

[0028] Furthermore, the structured prompts include the following functional modules: (1) Role setting module: The model is set as a visual question answering assistant for blind users. The input is declared as a first-person perspective video and a question. The task is to output structured evidence annotation results based only on the contextual visual evidence that is directly visible in the entire video, and finally make an answer. (2) Answer Principles Module: The model can only use information that is directly visible in the video frame and must not guess or fabricate missing information; the answer should meet the actual needs of blind users in real scenarios, provide truly useful information, and be easy for blind users to understand; unless it is directly related to the question, visual attributes that are useless to blind users should not be added. (3) Perspective and Expression Constraint Module: The constraint model makes spatial judgments from the user's first-person perspective by default, uses "you" as the subject when appropriate, and does not use bystander perspective expressions such as "in the video" or "in the scene". The spatial description is based on the scene state at the end of the video, i.e. the moment the user asks a question. (4) Uncertainty Constraint Module: The constraint model does not guess or pretend to know the answer when the video is insufficient to provide reliable visual evidence; even if the final answer cannot be determined, all directly visible relevant visual evidence must still be extracted and reported; when the answer is uncertain, the specific reasons for the uncertainty must be explained based on the actual situation of the video, such as the target being partially occluded, out of frame, too small, blurry, glare or reflection, only partially visible, or key details not clearly presented; it is forbidden to use general statements such as "the answer cannot be confirmed", but should briefly describe what can be seen and why the visible evidence is insufficient to determine the answer; (5) Annotation Requirements Module: Defines the structured fields of the model output and the value constraints of each field. The model must output a concise JSON object with the following field structure: { "evidence": [ { "frame_ids": [1, 2], "evidence_type": "object|text|spatial|quality|people|environment|other", "evidence_text": "short, objective, factual description of visiblecontent", "supports_answer": true } ... ], "answerability": "sufficient" or "insufficient", "confidence": 0~100, "answer": "final answer or explanation of what can be observed andwhy the answer cannot be determined" } (6) Evidence Requirements Module: Constrain visual evidence extraction and answerability determination to be performed as independent tasks: Regardless of whether the question is answerable, all directly visible relevant visual evidence must be extracted; even if there are multiple relevant visual evidence, the question may still be determined as “insufficient”; the evidence describes the visible content itself, not the inference or hypothesis; the support_answer field is set to true only when the evidence directly supports the final answer, and set to false when the evidence is relevant to the question but is insufficient to establish or confirm the answer; when the answerability is “insufficient”, it must include evidence explaining the visible content and why the answer cannot be determined. (7) Label description module: The confidence level is constrained to be an integer from 0 to 100, representing the degree of certainty that the question can be answered reliably based solely on visible evidence; when the video is blurry, obscured, missing frames, reflected, too far away, too close, or other visually unreliable conditions, the confidence level is reduced accordingly; and when the answerability is "insufficient", a statement of inability to confirm must be made in the answer field. (8) Output rules module: The constraint model only outputs valid JSON results, without outputting markdown tags, code blocks, comments or additional text. The evidence must be concise, objective and based on directly visible content, and does not describe the internal reasoning process.

[0029] Furthermore, when extracting visual evidence, the model determines the type of visual information required to answer the user's question. This type of visual information includes color, text, objects, images, icons, depth, distance, size, person's state, overall environmental information, location, orientation, and other information. The model matches the extracted visual evidence with these visual information types to support the calculation of the evidence sufficiency score in step S130. In an extended implementation, the structured prompt may further require the model to explicitly provide the identified visual information type field `required_visual_info_types` in the output to enhance the auditability of the output results.

[0030] In one specific implementation, the user question is input into the model along with the video using the following user prompt template: "Question: {question}, return only JSON results", where {question} is the natural language question posed by the user.

[0031] It should be noted that the structured prompts in this step predeclare the complete field structure of the model output, which includes, in addition to the visual evidence field, answerability, confidence, and response fields. The answerability and confidence fields are generated in step S130, and the response field is generated in step S140. That is, steps S120 to S140 are not three independent model calls, but rather a logical division of the same model inference output: under the constraints of the structured prompts, the model generates the data in an autoregressive chain according to the field order of "first outputting visual evidence, then outputting answerability and confidence, and finally outputting the response." Subsequent fields are conditional on the previously generated content, thus realizing an explicit inference chain of "evidence first, then judgment, and finally response" in a single inference.

[0032] In another implementation, steps S120 to S140 can also be completed by multiple model calls: in this case, the prompt words for each call only contain the output requirements of the current step, and the output result of the previous step is used as the input of the next step call. For example, the visual evidence extracted in step S120, together with the user's question, is input into the model to perform the answerability determination in step S130.

[0033] S130, Determining responsiveness based on visual evidence.

[0034] Specifically, based on the frame sequence and question obtained in step S110, and the visual evidence extracted in step S120, the model determines whether the current video is sufficient to correctly answer the user's question and outputs the answerability result. During the evaluation process, each piece of visual evidence is matched with the user's question and the type of visual information required to answer the question, and an evidence sufficiency score is calculated. ,in:

[0035] In the formula, Indicates the first One piece of visual evidence, This indicates a question raised by the user. This represents the set of visual information types required to answer a question. Indicates the first The degree to which visual evidence supports the questions and the visual information required to answer them. This indicates the corresponding weight.

[0036] When the sufficiency of evidence score Greater than or equal to the threshold When the answerability is sufficient, the sufficiency of evidence is judged as "adequate"; when the sufficiency of evidence score is... Less than the threshold When the responsiveness is deemed "insufficient", that is:

[0037] in, means "sufficient", It indicates "insufficiency".

[0038] In a preferred embodiment, the above-mentioned evidence sufficiency score is used to determine the sufficiency of evidence. With threshold The comparison and judgment process can be implicitly completed by a multimodal large model through end-to-end prompts. Specifically, a pre-constructed answer existence label space containing various video quality states is built, including eight categories: clearly visible answer, blurry due to motion or jitter, reflection, partial information loss, too far away, too close away, target occlusion, and others. After evaluating the support of visual evidence and questions through its internal attention mechanism, the multimodal large model directly maps the results to the label space. When the model outputs the label "clearly visible answer," it is equivalent to... The responsiveness is determined to be "sufficient"; when the output is other quality defect labels, it is equivalent to The responsiveness is determined to be "insufficient", and the mapped label is also output as the specific reason for insufficient evidence, which is used by step S140 to generate unconfirmable statements and shooting adjustment suggestions.

[0039] In summary, when there is sufficiently clear and complete evidence in the video to support the answer to the question, the answerability is judged as "sufficient"; when the video has serious blurring, occlusion, reflection, target out of frame, distance too far or too close, etc., making it impossible to accurately and completely identify the visual information required to answer the question, the answerability is judged as "insufficient".

[0040] Furthermore, the model can simultaneously output a confidence score, which represents the model's confidence level in whether the current video can correctly and safely answer the question. The confidence score is an integer from 0 to 100, positively correlated with the clarity, completeness, and reliability of visual evidence, and negatively correlated with problems such as blurriness, occlusion, glare, target outside the frame, and distance being too far or too close.

[0041] It's important to note that visual evidence extraction and answerability determination are independent tasks: even if the video quality is poor, rendering the question unanswerable, the model must still output the extracted visual evidence; furthermore, even if multiple pieces of relevant visual evidence exist, the question may still be deemed "insufficient." This mechanism ensures that the system output remains interpretable even in scenarios with insufficient evidence.

[0042] S140, generate a final answer or an unconfirmable statement based on responsiveness.

[0043] Specifically, when the responsiveness determination in step S130 is "sufficient," the model generates an accurate, direct, and user-friendly final answer based on visual evidence, and explicitly includes strict expression constraints in the prompts: spatial judgment must be made from the user's first-person perspective, using "you" as the subject, and unless the user explicitly asks a question, unnecessary visual features such as colors are prohibited in the answer. When the responsiveness determination is "insufficient," the model does not output speculative answers but instead outputs a statement of inconfirmation, which is a natural language output clearly stating that the current video cannot support an accurate answer. The statement of inconfirmation must include two parts: first, a description of the content actually visible in the current video; second, a specific reason for the inconfirmation based on the actual situation of the video, such as the target being partially obscured, out of frame, too small, blurry, having glare or reflection, only partially visible, or key details not clearly presented; it is prohibited to output only a general statement such as "inconfirmation" without specific reasons. For example, when dealing with the question "How much does this bottle of milk cost?", if there is glare in the price tag area, the model outputs an unconfirmed statement as "There is a price tag on the shelf, but the price cannot be confirmed due to glare," rather than the general "Unconfirmed."

[0044] Furthermore, the model generates a brief shooting adjustment suggestion based on the specific reason why the answer cannot be confirmed, so that the user can re-capture more complete and clearer visual information. In a preferred embodiment, the correspondence between the specific reason and the shooting adjustment suggestion includes: blurry or shaky image corresponds to suggesting that the user hold the device steadily and reduce shaking; too far away corresponds to suggesting that the user move closer to the target; too close away corresponds to suggesting that the user increase the distance from the target; reflection or glare corresponds to suggesting that the user adjust the shooting angle to avoid the light source or improve lighting; target is obscured or out of frame corresponds to suggesting that the user adjust the lens angle to make the target fully in the frame.

[0045] S150 outputs structured results.

[0046] Specifically, the visual evidence, answerability, confidence level, final answer, or unconfirmed statement output from steps S120 to S140 are uniformly organized into a structured result. This structured result can be saved in JSON or tabular format for subsequent data analysis, model fine-tuning, and automated evaluation.

[0047] In one specific implementation, when the responsiveness is determined to be "sufficient," the structured output result is shown in the following example: { "evidence": [ {"frame_ids": [3, 4], "evidence_type": "text", "evidence_text": "The shelf label shows a price of 5.99", "supports_answer": true}, {"frame_ids": [4], "evidence_type": "object", "evidence_text": "User holding a bottle of milk", "supports_answer": true} ], "answerability": "sufficient", "confidence": 92, Answer: "The price of this bottle of milk you have is $5.99." } When the responsiveness is determined to be "insufficient", the structured output result is shown in the following example: { "evidence": [ {"frame_ids": [6], "evidence_type": "quality", "evidence_text": "The price tag area is reflective, and the numbers are illegible", "supports_answer": false} ], "answerability": "insufficient", "confidence": 35, Answer: "There are price tags on the shelf, but the prices are unclear due to glare. We suggest adjusting the shooting angle slightly to avoid direct light." Furthermore, during the training phase, a strong multimodal model can be used as the teacher model to automatically generate visual evidence and answerable annotations for the training samples, and these structured annotations can be used to fine-tune the target model.

[0048] In summary, this embodiment provides a method for assistive video question answering for the blind based on visual evidence extraction and answerability determination. By guiding a multimodal large model to extract directly visible visual evidence from the video that can support the answer to the question, and then determining whether the visual evidence is sufficient to answer the question, a final answer or statement of inconclusiveness is generated. This effectively alleviates the dangerous hallucination problem of multimodal large models in scenarios with missing information or low-quality video, and improves the safety and reliability of assistive video question answering for the blind.

[0049] Example 2

[0050] This embodiment provides an experimental verification method for the above method on a public dataset to evaluate the effectiveness of the method in assistive video question answering tasks for the blind. It should be noted that the multimodal large model in this embodiment has not undergone any fine-tuning training for this task; that is, the method is implemented with zero samples, relying solely on structured prompts to guide the general multimodal large model through steps S110 to S150.

[0051] I. Experimental Setup

[0052] In one specific implementation, the multimodal large model uses Qwen3-VL-8B-Instruct (an open-source multimodal video understanding model with approximately 8 billion parameters) and employs the vLLM inference engine for batch inference. The main parameter configurations during inference are shown in the table below: Table 1. Main inference parameters of the Qwen3-VL-8B-Instruct model

[0053] Furthermore, the model's original output undergoes parsing and fault tolerance processing: First, any potential thought chain tags are removed, and Markdown code block markers are stripped away; then, an attempt is made to parse the output into a JSON object; if parsing fails, the content between the first "{" and the last "}" in the output is extracted and parsed again; if this still fails, the answerability is set to "insufficient", the confidence level is set to 0, and the answer is set to a preset catch-all text. This fault tolerance mechanism ensures the integrity of the structured output fields and avoids missing system output due to parsing anomalies.

[0054] II. Experimental Data

[0055] In one specific implementation, the experiment used the test set of the publicly available first-person perspective video question-answering dataset for the blind, VisAssist, from which 400 samples were randomly selected. Each sample included: a first-person perspective video taken by a blind user, an English question, one or more English reference answers, an answer presence label, and a visual information type label. The answer presence label indicated whether the answer was visible in the video and the type of quality defect when it was not visible, including eight categories: clearly visible answer, blurring due to motion or shakiness, reflection, missing information, distance too far, distance too close, target occlusion, and others. The visual information type label indicated the type of visual information relied upon for answering the question, including nine categories: color, text, objects, images / icons, depth, distance, size, person's state, overall environmental information, location and direction, and others. Based on the answer presence label, the samples were divided into a "visible answer" group and a "not visible answer" group to examine the model's performance in scenarios with sufficient and insufficient evidence, respectively.

[0056] III. Evaluation Methods

[0057] The large-scale model automatic evaluation method is adopted. The question, reference answer, and predicted answer of the model to be evaluated for each sample are input into the large-scale evaluation model, which then gives an integer score from 0 to 5 in the following three dimensions: (1) Factual accuracy cor: assess whether the predicted answer and the reference answer are consistent in factual terms, whether there is any misinterpretation or false information, and allow synonym substitution and rewriting; (2) Completeness of details: Evaluate the extent and specificity of the predicted answer's coverage of key points in the video; (3) Spatial understanding su: assess the accuracy of the predicted answer in describing spatial relationships such as location, direction, and layout in the video.

[0058] Furthermore, the average score of the three dimensions, avg = (cor + do + su) / 3, is taken as the comprehensive score for a single sample, and the average scores are calculated separately for all samples and for each label group. For samples with multiple reference answers, the optimal result compared with each reference answer is taken. In one specific implementation, the evaluation model uses the DeepSeek-chat model, where the evaluation model is called independently for each of the three dimensions and constrained to return only integer scores from 0 to 5.

[0059] IV. Comparison Settings

[0060] To verify the contribution of each constraint module in the structured prompt word of this invention, the following comparison group was set up: Group A: The complete structured prompt of this invention, that is, including all the functional modules described in Embodiment 1; Group B: Based on the structured prompts, this group retains only the role setting, answer principles, perspective and expression constraints, and basic output rules, but does not include the uncertainty constraints, annotation requirements, evidence requirements, and label description modules.

[0061] Both groups used the same base model, the same test set, and the same inference parameters. The only variable was the content of the cue words, thus examining the impact of each constraint module in the cue words on the model behavior separately.

[0062] V. Experimental Results

[0063] The overall evaluation results of the two sets of settings on the test set are shown in the table below: Table 2 Evaluation results of Group A (complete structured prompts of this invention) and Group B (control prompts)

[0064] As shown in Table 2, in terms of overall average score (avg), group A (2.5667) improved by approximately 9.6% compared to group B (2.3425). Among the three dimensions, the improvement in factual accuracy (cor) was the most significant (2.6575 vs. 2.2175, an improvement of approximately 19.8%), indicating that the evidence requirements and uncertainty constraints in the structured prompts effectively reduced factual errors in the model's output. Further group analysis revealed that in the "Answer Visible" group, the overall scores of the two groups were basically the same (2.4150 vs. 2.3955), indicating that when video evidence is sufficient, the prompts can guide the model to make basically correct answers. However, in the "Answer Not Visible" group, the overall score of group A (3.8943) was much higher than that of group B (1.8780), an improvement of approximately 107%. It should be noted that the reference answer for the "answer not visible" group is itself a description of the situation where the answer cannot be confirmed. Therefore, in this group, answers that truthfully output the statement of inconfirmation and the specific reason are more consistent with the reference answer, while speculative answers are seriously inconsistent with the reference answer. The significant lead of group A in this group indicates that the uncertainty constraint, evidence requirement, and label description modules can significantly suppress speculative answers by the model when there is insufficient evidence. The above results show that the chain mechanism of "evidence first, judgment second, and answer last" of this invention significantly improves the security and reliability of the output in the case of insufficient evidence without significantly affecting the answer quality in the case of sufficient evidence, which is consistent with the technical problem to be solved by this invention.

[0065] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A video-assisted question-answering method for the blind based on visual evidence extraction and responsiveness assessment, characterized in that, Includes the following steps: S110: Acquire video input and user questions, and perform preprocessing; receive first-person perspective video taken by a blind user and the natural language questions raised, decode the video and sample frames to obtain a frame sequence, and normalize the size of the sampled frames; S120, Based on structured prompts, guide the multimodal large model to extract visual evidence; input the frame sequence and user question into the multimodal large model, guide the model to identify the type of visual information required to answer the question through structured prompts, and extract directly visible and question-related visual evidence from the video frames, while generating a visual evidence description containing frame-level positioning information; the structured prompts include at least the following constraints: extraction is based only on information directly visible in the video frames, and speculative evidence or common-sense supplementation is prohibited for content that cannot be directly confirmed from the video; S130, Determine answerability based on visual evidence; match the extracted visual evidence with the user's question, calculate the evidence sufficiency score, and determine whether the answerability is "sufficient" or "insufficient" based on the comparison result of the score with a preset threshold; at the same time, the model outputs an answerability confidence score that is positively correlated with the sufficiency of evidence and negatively correlated with video quality issues; the answerability determination is independent of the visual evidence extraction, and even if the visual evidence is insufficient to support the answer, the extracted visual evidence is still output; S140, generate a final answer or an unconfirmed statement based on answerability; if answerability is determined to be "sufficient", generate a natural language answer based on the question and visual evidence; if answerability is determined to be "insufficient", generate a statement that the answer to the current question cannot be confirmed, and generate a shooting adjustment suggestion to obtain clearer visual information. S150, outputs structured results; Organize and output the visual evidence, answerability determination results, confidence levels, and final answers or statements of nonconfirmation into a structured result in JSON or tabular format.

2. The assistive video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in claim 1, characterized in that, Step S110, which involves acquiring video input and user questions and performing preprocessing, specifically includes: S1101, receiving a first-person perspective video taken by a blind user and the natural language questions they raise; the natural language questions are obtained through text input, voice input transcribed by speech recognition, or synchronous questioning during video recording. S1102, after decoding the video, frame sampling is performed at a fixed frame rate, and the number of sampled frames does not exceed the preset frame number limit; when the number of frames obtained by sampling at the fixed frame rate exceeds the preset frame number limit, the video is uniformly sampled to ensure that the number of frames in the final input model does not exceed the preset frame number limit. S1103, normalize the size of the sampled frame: while maintaining the aspect ratio, scale the sampled frame so that the total number of pixels in a single frame image is between the preset lower limit and the preset upper limit of pixels.

3. The assistive video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in claim 2, characterized in that, The fixed frame rate is preferably 0.5fps to 1fps, and the preset frame rate upper limit is preferably 10 to 20 frames.

4. The assistive video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in claim 2, characterized in that, The preset lower pixel limit is preferably 32×32 pixels, and the preset upper pixel limit is preferably 336×336~448×448 pixels.

5. The assistive video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in claim 1, characterized in that, The visual evidence extraction described in step S120 specifically includes: S1201, Construct structured prompts, which include the following functional modules: role setting module, answer principle module, perspective and expression constraint module, uncertainty constraint module, annotation requirement module, evidence requirement module, label description module, and output rule module; S1202, identify the type of visual information required to answer the question, and determine the type of visual information it depends on based on the question. The type of visual information includes color, text, object, image icon, depth distance size, person status, overall environment information, position and direction, and other information. S1203, Input the frame sequence and user question into the multimodal large model, and identify visual evidence that is directly visible and relevant to the question from the video frames; The model only retains visual evidence that can be directly seen from the video and can support the answer, and must not generate speculative evidence or common sense supplements for content that cannot be directly confirmed from the video. S1204, Generate visual evidence description and frame-level positioning information. Provide a brief text description for each piece of visual evidence and mark the corresponding frame number or frame interval. Each piece of visual evidence includes at least a frame number or frame interval field, an evidence type field, an evidence text description field, and a support marker field. The evidence type includes objects, text, spatial location, image quality, people, environment, and others. When the visual evidence is related to the question but is insufficient on its own to establish or confirm the answer, the support marker field is set to no.

6. The assistive video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in claim 1, characterized in that, Step S130, which involves determining responsiveness based on visual evidence, specifically includes: S1301, Determine whether the visual evidence extracted in step S120 is sufficient: Match each piece of visual evidence with the user's question and the type of visual information required to answer the question, and calculate the sufficiency score of the evidence. , ,in, Indicates the first One piece of visual evidence, This indicates a question raised by the user. This represents the set of visual information types required to answer a question. Indicates the first The degree of support of visual evidence for answering the question. Indicates the corresponding weight. The number of visual evidence items; when the sufficiency of evidence score is... Greater than or equal to the preset threshold When the answerability is sufficient, the sufficiency of evidence is judged as "adequate"; when the sufficiency of evidence score is... Less than the preset threshold In this case, the responsiveness is judged as "insufficient"; S1302, the calculation and threshold comparison process of the evidence sufficiency score is implicitly completed by the multimodal large model under the guidance of structured prompts: a pre-constructed answer existence label space is built, which includes the following: answer clearly visible, blurry due to motion or shaking, reflection, partial information loss, distance too far, distance too close, target occlusion, and others; the model maps the answerability judgment result to the answer existence label space, and when the mapping result is "answer clearly visible", it is equivalent to... The responsiveness is determined to be "sufficient"; when the mapping result is another label, it is equivalent to The answerability is judged as "insufficient", and the mapped label is output as the specific reason for insufficient evidence; S1303, output answerability confidence score. The model gives an integer score of 0 to 100 for "whether the current video can answer the question correctly and safely", which represents the degree of confidence that the question can be answered reliably based solely on visible evidence. The confidence score is positively correlated with the sufficiency of evidence and negatively correlated with problems such as blurriness, reflection, occlusion, target out of frame, and distance being too far or too close.

7. The assistive video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in claim 1, characterized in that, Step S140, which involves generating a final answer or an unconfirmable statement based on answerability, specifically includes: S1401, if the answerability determination in step S130 is "sufficient", then generate a natural language answer that is understandable and operable for the blind user based on the question and visual evidence; the natural language answer is expressed from the user's first-person perspective, with "you" as the subject, and does not contain visual attributes that are not directly related to the question unless the user explicitly asks in the question. S1402, if the answerability determination in step S130 is "insufficient", the model does not output a speculative answer, but generates a statement that the answer to the current question cannot be confirmed based on the question and the missing visual information; the statement that the answer cannot be confirmed describes the content that is actually visible in the current video, and explains the specific reasons for the inability to confirm the answer based on the actual situation of the video. The specific reasons include the target being occluded, the target being out of the frame, the target being too small, the image being blurry, reflection or glare, the target being only partially visible or key details not being clearly presented; it is prohibited to output only a general statement that the answer cannot be confirmed without specific reasons; S1403, Based on the specific reason why the answer cannot be confirmed, generate a brief shooting adjustment suggestion, which includes adjusting the lens angle, moving closer to the target, increasing the distance, reducing shaking, improving lighting, or avoiding reflections, in order to re-acquire more complete and clearer visual information.

8. The assistive video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in claim 1, characterized in that, The output structured result described in step S150 specifically includes: S1501, when the responsiveness of step S130 is determined to be "sufficient", the visual evidence, the responsiveness determined to be "sufficient", the confidence level and the final answer are organized into a structured output result. S1502, when the responsiveness of step S130 is determined to be "insufficient", the visual evidence, the responsiveness of the "insufficient" determination, the confidence level, the reason for the insufficiency, the statement that cannot be confirmed, and the shooting adjustment suggestions are organized into a structured output result. S1503, the structured results are saved in JSON or tabular format, and include at least a visual evidence list field, an answerability field, a confidence field, and an answer field; wherein, the answerability field takes the value "sufficient" or "insufficient", and the confidence field takes the value of an integer from 0 to 100; the structured results are used for subsequent data analysis, or as structured annotations during the training phase for fine-tuning the target model.

9. A video-assisted question-and-answer device for the blind based on visual evidence extraction and responsiveness determination, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the video question-answering method for the blind based on visual evidence extraction and responsiveness determination as described in any one of claims 1-8.

Images