Visual perception task processing method and device, storage medium and computer equipment

Abstract

The invention relates to the technical field of artificial intelligence and computer vision, and discloses a visual perception task processing method and device, a storage medium and computer equipment, and the method comprises the steps: generating a plurality of corresponding candidate visual inputs in response to a received visual perception task request comprising visual inputs and user query; inputting the candidate visual input and the user query into a multi-modal large language model, calculating an uncertainty score, and screening out at least one optimal candidate visual input; and inputting the optimal candidate visual input and the user query into the multi-modal large language model, and generating and outputting a response result for the visual perception task request. According to the method, through a unified framework which does not need training, only utilizes internal uncertainty signals of the model and keeps end-to-end processing, the fine-grained sensing ability of the model is improved while resource consumption is reduced, and the practicability and reliability in high-difficulty fine-grained sensing tasks such as tiny focus areas in medical images are enhanced.

Description

Technical Field

[0001] This invention relates to the technical fields of artificial intelligence and computer vision, and in particular to a visual perception task processing method, apparatus, storage medium, and computer equipment. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, multimodal large language models have demonstrated powerful capabilities in tasks such as general visual scene understanding and question answering. As the foundation of cross-modal intelligence, they have broad application potential in fields such as medical image analysis where the requirements for perception accuracy and decision reliability are stringent. By integrating visual and linguistic information, these models can perform holistic description and reasoning of complex scenes, providing new technical paths for tasks such as image report generation and multimodal medical record understanding in the medical field. However, the current processing paradigm of these models exhibits significant limitations when faced with massive, high-resolution, or long-term input data, restricting their ability to play a key role in professional scenarios that require precise and fine-grained perception.

[0003] Specifically, existing technologies still have shortcomings when dealing with complex visual processing tasks involving fine-grained and localized perception. For example, in medical image analysis, it is necessary to accurately locate and describe tiny lesion areas from a whole high-resolution pathological slice. Current mainstream solutions mainly rely on two paths: one is to perform large-scale supervised fine-tuning or reinforcement learning training on the base model for specific tasks to learn task-specific attention patterns or output formats; the other is to introduce and integrate external dedicated models, such as object detectors or segmentation networks, to provide localization information. Although these methods can improve the performance indicators of specific tasks, they have significant drawbacks. Specifically, on the one hand, the computational cost is high and the engineering deployment process is complex, making it difficult to quickly promote and iterate in fields such as medicine where there are strict requirements for data privacy, system stability, and real-time performance. On the other hand, they are highly dependent on task-specific data and design, lacking generality between tasks. At the same time, integrating external modules destroys the simplicity and internal consistency of the end-to-end system and may introduce new error propagation and coordination overhead.

[0004] Therefore, a visual perception task processing method that can be widely applied, requires no additional training costs, and effectively guides the model to complete fine perception tasks in massive input information is urgently needed. Summary of the Invention

[0005] In view of this, this application provides a visual perception task processing method, apparatus, storage medium and computer equipment, the main purpose of which is to solve the technical problems of high computational training cost, complex process and poor versatility and general applicability of the prior art when facing complex visual processing tasks.

[0006] According to a first aspect of the present invention, a visual perception task processing method is provided, comprising: In response to a received visual perception task request, multiple candidate visual inputs corresponding to the visual perception task request are generated, wherein the visual perception task request includes visual inputs and user queries; The candidate visual inputs and the user query are respectively input into a preset multimodal large language model, and the uncertainty score used to characterize the multimodal large language model's response to the user query is calculated. Based on the uncertainty score, at least one optimal candidate visual input is selected from the plurality of candidate visual inputs; The optimal candidate visual input and the user query are then input into the multimodal large language model to generate and output a response result for the visual perception task request.

[0007] Optionally, generating multiple candidate visual inputs corresponding to the visual perception task request includes: when the visual input is an image, cropping the image based on multiple preset spatial scales and / or multiple spatial locations to generate multiple image cropping blocks as candidate visual inputs; when the visual input is a video, sampling the video based on multiple preset time intervals and / or multiple time points to generate multiple video segments and / or video frames as candidate visual inputs.

[0008] Optionally, the calculation of the uncertainty score characterizing the multimodal large language model's response to the user query includes: obtaining the answer sequence generated by the multimodal large language model in response to the user query, wherein the answer sequence includes multiple tokens; obtaining the probability distribution of the tokens on the vocabulary of the multimodal large language model, and calculating the Shannon entropy of the tokens based on the probability distribution; calculating the arithmetic mean of the Shannon entropies of all tokens in the answer sequence, and using the arithmetic mean as the uncertainty score.

[0009] Optionally, the calculation used to characterize the uncertainty score of the multimodal large language model in response to the user query includes: constructing a binary judgment question related to the user query based on the candidate visual input; inputting the candidate visual input and the binary judgment question into the multimodal large language model, obtaining the confidence score for the affirmative answer from the output of the multimodal large language model; performing a negative correlation transformation on the confidence score to obtain a transformation value, and using the confidence score or the transformation value as the uncertainty score corresponding to the candidate visual input.

[0010] Optionally, the step of selecting at least one optimal candidate visual input from multiple candidate visual inputs based on the uncertainty score includes: when the multiple candidate visual inputs are arranged in chronological order, forming a confidence sequence by arranging the uncertainty scores corresponding to the candidate visual inputs in the order of the multiple candidate visual inputs; determining a continuous optimal subsequence in the confidence sequence by solving the maximum subarray sum algorithm, and using the continuous time window corresponding to the optimal subsequence as the optimal candidate visual input.

[0011] Optionally, the step of re-inputting the optimal candidate visual input and the user query into the multimodal large language model to generate and output a response result for the visual perception task request includes: inputting the continuous time window corresponding to the optimal subsequence and the user query into the multimodal large language model to obtain the natural language response generated by the multimodal large language model, wherein the natural language response is used to describe the target event content; mapping the start and end positions of the continuous time window back to the temporal coordinates of the visual input to obtain the start and end timestamps of the target event content; generating and outputting a response result for the visual perception task request, wherein the response result includes the natural language response, the start timestamp, and the end timestamp.

[0012] Optionally, after selecting at least one optimal candidate visual input from among the multiple candidate visual inputs, the method further includes: obtaining contextual visual information adjacent to the optimal candidate visual input based on the temporal or spatial attributes of the optimal candidate visual input; fusing the contextual visual information with the optimal candidate visual input to form an enhanced visual input; and using the enhanced visual input as the optimal candidate visual input for generating the response result.

[0013] According to a second aspect of the present invention, a visual perception task processing apparatus is provided, comprising: A candidate generation module is used to generate multiple candidate visual inputs corresponding to a received visual perception task request, wherein the visual perception task request includes visual inputs and user queries. The candidate evaluation module is used to input multiple candidate visual inputs and the user query into a preset multimodal large language model, and calculate the uncertainty score used to characterize the multimodal large language model's response to the user query. A candidate filtering module is used to filter at least one optimal candidate visual input from a plurality of candidate visual inputs based on the uncertainty score; The result output module is used to input the optimal candidate visual input and the user query back into the multimodal large language model to generate and output the response result for the visual perception task request.

[0014] According to a third aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described visual perception task processing method.

[0015] According to a fourth aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described visual perception task processing method.

[0016] This invention provides a visual perception task processing method, apparatus, storage medium, and computer device. First, it eliminates the need for any parameter updates or task-specific training of existing multimodal large language models. The model is dynamically guided through a process of candidate generation, score calculation, optimal selection, and re-response. All calculations are performed based on a pre-trained model, reducing application costs and deployment barriers. Second, it utilizes the inherent signals of the multimodal large language model itself—the uncertainty score corresponding to its output response—as the decision-making basis. The entire process interacts only with a single multimodal large language model, without integrating any external modules, thus maintaining end-to-end simplicity and internal consistency, and reducing the risk of error propagation due to differences between modules. Finally, by actively generating multiple candidate visual inputs and evaluating their uncertainty, it essentially guides the model to perform an adaptive, data-driven focus search on massive amounts of input information. This effectively improves the model's ability to complete highly complex, fine-grained perception tasks without retraining, enhancing its practicality and reliability in professional scenarios. The above method improves the model's fine-grained perception capability while reducing resource consumption by using a unified framework that requires no training, utilizes only the model's inherent uncertainty signals, and maintains end-to-end processing. This enhances its practicality and reliability in challenging fine-grained perception tasks such as small lesion areas in medical imaging.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 A flowchart illustrating a visual perception task processing method provided by an embodiment of the present invention is shown. Figure 2 A flowchart illustrating another visual perception task processing method provided by an embodiment of the present invention is shown; Figure 3 This diagram illustrates the structure of a visual perception task processing device according to an embodiment of the present invention. Figure 4 This invention provides a schematic diagram of the structure of another visual perception task processing device according to an embodiment of the invention. Figure 5 A schematic diagram of the device structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0020] This application provides a method for processing visual perception tasks, such as... Figure 1 As shown, the method includes the following steps: 101. In response to the received visual perception task request, generate multiple candidate visual inputs corresponding to the visual perception task request, wherein the visual perception task request includes visual inputs and user queries.

[0021] In this context, a visual perception task request refers to a specific instruction encapsulated to understand and analyze visual content. It consists of visual input and user query. Visual input is raw visual data such as images and video sequences, while user query is natural language text expressing specific needs. In the context of medical applications, visual input can be a high-resolution pathological slide image, and user query can be a doctor's request to "locate and describe the suspicious cell region in the image." Candidate visual input refers to multiple subsets or perspectives that may contain key information, which the system actively derives from the raw visual input in response to this request, such as different local image patches cropped from the whole slide.

[0022] Specifically, the visual perception task processing method disclosed in this application does not directly hand over the original and complete visual data to the model for processing. Instead, it first strategically constructs a diverse set of candidate observations based on the semantics of the task request and the structure of the visual content. For medical images, this step can be manifested as generating a series of image cropping blocks covering different tissue parts and having different magnification ratios if the input is a static pathological image. These candidate visual inputs aim to systematically cover various parts or moments in the original data that may contain key information, laying the foundation for subsequent detailed evaluation.

[0023] In this embodiment of the application, by generating and introducing multiple focused candidate views, a mechanism is essentially provided for the model to go beyond the global and coarse processing of the original massive data and instead have the opportunity to conduct in-depth analysis of smaller and more relevant data units. Without the need for targeted training of the model or integration of external detectors, the model can be guided to adaptively focus its attention on potentially abnormal local regions.

[0024] 102. Input multiple candidate visual inputs and user queries into a pre-defined multimodal large language model, and calculate the uncertainty score used to characterize the multimodal large language model's response to the user query.

[0025] The pre-defined multimodal large language model refers to a general-purpose base model that has been pre-trained on large-scale cross-modal data and has the ability to understand and generate. In the medical context, it can interpret images and answer related medical questions. The uncertainty score is a quantitative indicator used to measure the certainty or confidence of the model's generated answer when given a specific candidate visual input and user query. The lower the score, the more certain the model is about to generate the correct answer. The higher the score, the more hesitant the model is. This score does not depend on external tools and is derived entirely from the model's internal state during the reasoning process.

[0026] Specifically, each candidate visual input generated in the first stage is paired with the original user query and then sequentially input into the multimodal large language model. For each pair of inputs, the model attempts to generate an answer. The system does not directly adopt this answer, but analyzes the probability distribution output by the model during the answer generation process and calculates a unified uncertainty score. This process is equivalent to allowing the model to evaluate the degree of confidence in responding to the candidate visual inputs, and finally obtains an uncertain quantitative index.

[0027] In this embodiment, the model's own intrinsic cognitive signals are discovered and utilized as the basis for decision-making. For example, when the model observes an image patch containing typical and clear lesion features, the probability of generating a confirmed diagnosis will be highly concentrated, resulting in a very low uncertainty score. Conversely, when faced with blurry and normal tissue, the model output will appear hesitant, and the score will be higher. This provides an objective and quantifiable data foundation for subsequent screening of the most relevant evidence, without the need to introduce external models or conduct specific task training, thus providing technical support for achieving efficient and adaptive medical image analysis.

[0028] 103. Based on the uncertainty score, select at least one optimal candidate visual input from multiple candidate visual inputs.

[0029] Among them, the optimal candidate visual input refers to the subset of visual data that, after quantitative evaluation, is determined to be the most likely to contain the key information needed to answer the user's query. Specifically, it is determined based on the uncertainty score calculated in the previous step. The candidate with the lowest score means that the multimodal large language model has the highest certainty in answering it, and therefore is logically regarded as the most relevant and highest quality evidence.

[0030] Specifically, based on the uncertainty score calculated for each candidate visual input in the previous stage, the system executes a clear optimization selection strategy, such as directly selecting one or more candidates with the lowest scores. For example, in the analysis of pathological slides, the system has calculated uncertainty scores for dozens of cropping blocks at different locations and scales, and selects one or several image blocks with the highest scores through simple sorting and comparison.

[0031] In this embodiment, data-driven, automated, and objective attention focus is achieved, fully trusting and utilizing the deterministic differences exhibited by the model itself to make decisions. This means that the system can automatically and without human intervention focus subsequent analysis resources and attention on the image areas most likely to contain decisive diagnostic evidence. This not only improves the automation and intelligence of the processing flow but also lays a solid foundation for generating reliable and accurate response results.

[0032] 104. Input the best candidate visual input and the user query back into the multimodal large language model to generate and output the response results for the visual perception task request.

[0033] The response result is a complete and direct answer generated by a multimodal large language model based on the best visual evidence provided to the user query. In medical tasks, this result may be a description of a selected image region, such as a region showing spiculated edges and high-density shadows typical of malignant tumors, or a summary diagnostic opinion combined with specific findings.

[0034] Specifically, one or more of the selected optimal candidate visual inputs are used as the sole or primary visual context, which is then combined with the initial user query to form a concise and focused question-and-answer request. This request is then submitted to a multimodal large language model. For example, when analyzing medical images, the system no longer displays the entire CT scan image, but instead inputs only the cropped region containing the most suspicious nodule, along with instructions describing the lesion, into the model. Based on the purified, highly relevant information, the model executes its core generative capabilities and outputs the final text answer.

[0035] In this embodiment, by precisely guiding the model's attention to the most relevant information, the ambiguity caused by redundant or irrelevant input information is effectively suppressed, thereby significantly improving the accuracy and reliability of the answer. In the context of medical auxiliary analysis, this means that the description or inference given by the system is closely related to the most diagnostically valuable image region, significantly enhancing the clinical reference value and trustworthiness of the results. At the same time, since the final inference only needs to process a small amount of carefully selected data, efficiency is also optimized, thus constructing a real-time and accurate clinical decision support tool.

[0036] This invention provides a visual perception task processing method. First, it eliminates the need for any parameter updates or task-specific training of existing multimodal large language models. The method dynamically guides the model through a process of candidate generation, score calculation, optimal selection, and re-response. All calculations are performed based on a pre-trained model, reducing application costs and deployment barriers. Second, it utilizes the inherent signals of the multimodal large language model itself—the uncertainty score corresponding to its output response—as the decision-making basis. The entire process interacts only with a single multimodal large language model, without integrating any external modules, thus maintaining end-to-end simplicity and internal consistency of the system and reducing the risk of error propagation due to differences between modules. Finally, by actively generating multiple candidate visual inputs and evaluating their uncertainty, it essentially guides the model to perform an adaptive, data-driven focus search on massive amounts of input information. This effectively improves the model's ability to complete highly complex, fine-grained perception tasks without retraining, enhancing its practicality and reliability in professional scenarios. The above method improves the model's fine-grained perception capability while reducing resource consumption by using a unified framework that requires no training, utilizes only the model's inherent uncertainty signals, and maintains end-to-end processing. This enhances its practicality and reliability in challenging fine-grained perception tasks such as small lesion areas in medical imaging.

[0037] This application provides another method for visual perception task processing, such as... Figure 2 As shown, it specifically includes: 201. In response to the received visual perception task request, generate corresponding candidate visual inputs based on different visual input types.

[0038] Specifically, when the visual input is an image, the image is cropped based on multiple preset spatial scales and / or multiple spatial locations to generate multiple image cropping blocks as candidate visual inputs; when the visual input is a video, the video is sampled based on multiple preset time intervals and / or multiple time points to generate multiple video segments and / or video frames as candidate visual inputs.

[0039] In this embodiment, visual input can be mainly divided into two categories: static images and dynamic videos. The candidate generation strategies differ for different types. When the visual input is a static image, such as a pathological slide or X-ray in medicine, the system processes the original image according to preset diverse strategies to generate a series of representative local views, i.e., image cropping blocks. One processing method considers the diversity of spatial scales, i.e., cropping according to different scaling ratios or viewing angles to simulate the process of zooming in or out to observe details. Another processing method considers the diversity of spatial positions, i.e., using methods such as sliding windows, grid division, or interest point detection to ensure that the cropping blocks cover different areas of the image. By combining different scales and positions, a set of candidate visuals can be systematically generated. The system takes input data to ensure that regardless of the scale and location of the target object or region in the image, there are corresponding candidate cropping blocks to include it. When the visual input is dynamic video, such as an echocardiogram, the system generates candidate visual input based on its time dimension. This also includes two main methods: one is to sample based on multiple time intervals, such as extracting a frame every 1 second, 5 seconds, or 10 seconds, or to perform sliding sampling with a fixed duration window to generate a series of short video clips; the other is to extract based on multiple time points, such as selecting frames corresponding to moments in the video where the content changes significantly through a keyframe detection algorithm. The video clips or video frames generated in this way are the candidate visual inputs for the video input, collectively covering different key moments or time periods in the video stream.

[0040] Based on this step, the original visual input, which may contain massive amounts of information, is transformed into a limited set of candidate visual inputs that systematically covers multiple scales, locations, or times. This set serves as the basis for subsequent uncertainty assessment, allowing the model to evaluate pre-generated, more focused candidate units one by one instead of directly processing the complex original data.

[0041] 202. Input multiple candidate visual inputs and user queries into a pre-defined multimodal large language model, and calculate the uncertainty score of the multimodal large language model's response to the user query based on the probability distribution of the tokens.

[0042] Specifically, the system obtains the answer sequence generated by the multimodal large language model in response to user queries, wherein the answer sequence includes multiple tokens; obtains the probability distribution of the tokens on the vocabulary of the multimodal large language model, and calculates the Shannon entropy of the tokens based on the probability distribution; calculates the arithmetic mean of the Shannon entropy of all tokens in the answer sequence, and uses the arithmetic mean as the uncertainty score.

[0043] In this implementation, the model's internal probability distribution during answer generation is used to quantify the certainty of its output, thereby evaluating the relevance between the provided candidate visual inputs and the user query. This quantification metric is called Token Entropy. The calculation process includes: First, for each candidate visual input, it is input along with the user query into a multimodal large language model, and the complete answer generated by the model is obtained. This answer is represented internally by a series of discrete semantic units, i.e., tokens. These tokens together constitute an answer sequence. For example, for the query "Describe the organizational features of this region," the model may generate a text answer composed of multiple words or sub-word units. Next, the system delves deeper into the model to obtain the probability information at each step in the answer sequence generation process. Specifically, for each token position in the answer sequence, the model outputs a probability distribution over the entire vocabulary. The probability distribution reflects the confidence level of the model in selecting each possible word at the current position. Based on this probability distribution, the Shannon entropy of the token at this position can be calculated. Shannon entropy is a classic metric in information theory used to measure the uncertainty of a probability distribution. If the model is very certain about the token to be generated, its probability distribution will be highly concentrated on a specific word, resulting in a low entropy value. Conversely, if the model is hesitant, the probability distribution tends to be uniform, resulting in a high entropy value. Then, the Shannon entropy calculated for each token position in the answer sequence is accumulated and divided by the total length of the sequence, i.e., the total number of tokens, to obtain the arithmetic mean. This mean is the final uncertainty score for the candidate visual input. The uncertainty score measures the average degree of uncertainty of the model when generating the entire answer based on the current visual evidence.

[0044] The above calculation process can be formally expressed as the following formula:

[0045] In the formula, v represents the candidate visual input; q represents the user query; represents the average token entropy calculated given candidate visual input v and user query q, i.e., the uncertainty score; T represents the length of the answer sequence generated by the model, i.e., the total number of tokens contained in the answer; N represents the size of the multimodal large language model vocabulary, i.e., the number of all different tokens that the model can predict; p i,jThis represents the probability that, when generating the i-th token in the answer sequence, the model predicts the j-th token in the vocabulary as the current output. The value is between 0 and 1, and for a fixed i, the probability of all j corresponding to p... i,j The sum is 1.

[0046] 203. Input multiple candidate visual inputs and user queries into a preset multimodal large language model, and calculate the uncertainty score of the multimodal large language model's response to the user query based on a binary judgment problem.

[0047] Specifically, based on candidate visual input, a binary judgment question related to the user query is constructed; the candidate visual input and the binary judgment question are input into a multimodal large language model, and the confidence score for the affirmative answer is obtained from the output of the multimodal large language model; the confidence score is negatively correlated to obtain the transformed value, and the confidence score or the transformed value is used as the uncertainty score corresponding to the candidate visual input.

[0048] In this implementation, a binary judgment question with a yes or no answer is actively constructed, which is closely related to the original user's query intent. For example, when faced with a user querying whether there are signs of malignant tumors in an image, the system can construct a specific binary question, such as whether this image region displays typical features of malignant tumors, for each candidate pathological image crop. Then, the candidate visual input and this binary judgment question are input together into a multimodal large language model. When the model processes such questions, its output usually includes the probability or confidence level assignment for affirmative and negative answers. This method extracts the confidence level for affirmative answers from the model output. The confidence level value directly reflects the degree of certainty that the model gives an affirmative judgment on the binary question based on the currently seen visual evidence.

[0049] Furthermore, to maintain consistency with the definition of uncertainty score—that is, a lower score indicates higher certainty—the obtained confidence score needs to be negatively correlated. For example, if the model outputs a confidence score of 0.9 for "yes," its negatively correlated value can be directly set to 0.1. Alternatively, other monotonically decreasing functions can be used to process the score, and the converted value or the original confidence score itself can be used as the uncertainty score corresponding to the candidate visual input. Through this step, the system can quickly obtain a scalar score to measure the model's confidence in making a clear and affirmative judgment based on the candidate input. In medical image analysis, this method can efficiently screen out regions from a large number of candidate regions that the model is highly confident in the presence of specific pathological manifestations, providing a computationally lightweight and directly effective technique for rapid initial screening or focal localization.

[0050] 204. Select the candidate visual input with the lowest uncertainty score as the optimal candidate visual input.

[0051] Specifically, among multiple candidate visual inputs, one or more candidate visual inputs with the lowest uncertainty score are selected as the optimal candidate visual inputs.

[0052] In this implementation, after the system calculates the corresponding uncertainty scores for all candidate visual inputs, it sorts and compares them based on the score values. The candidate with the lowest uncertainty score means that the multimodal large language model exhibits the highest certainty or the lowest perplexity when responding to user queries based on visual evidence. Therefore, one or more candidates with the lowest scores are selected from the entire candidate set and formally defined as the optimal candidate visual input in this task processing flow. In the application scenario of medical image analysis, when analyzing a pathological whole slice image containing a large number of tissue structures, the system may generate dozens of image blocks covering different parts and different magnifications as candidates in the early stage. Each image block will obtain an uncertainty score for the model's response to it. Through this step, the system can automatically and accurately identify the image area that makes the model most confident in judging whether there is a lesion, and quickly lock the diagnostic focus on the area with the most diagnostic value and the clearest features, thereby providing a reliable guarantee for the subsequent generation of accurate reports.

[0053] 205. Arrange multiple candidate visual inputs in chronological order to form a confidence sequence, and select the optimal candidate visual input.

[0054] Specifically, when multiple candidate visual inputs are arranged in chronological order, the uncertainty scores corresponding to the candidate visual inputs are arranged in the order of the multiple candidate visual inputs to form a confidence sequence. By solving the maximum subarray sum algorithm, the continuous optimal subsequence is determined in the confidence sequence, and the continuous time window corresponding to the optimal subsequence is taken as the optimal candidate visual input.

[0055] In this implementation, the system first ensures that the candidate visual inputs generated by the system are arranged in their natural chronological order. After calculating the uncertainty score for each candidate visual input, the system arranges these scores in sequence to form a confidence waveform that changes over time. Then, the system applies the maximum subarray sum algorithm to this confidence sequence, with the goal of finding a continuous segment in the sequence whose sum of values ​​reaches the global maximum. This continuous segment is then determined as the optimal subsequence. Finally, the system maps the start and end indices of the optimal subsequence in the confidence sequence back to the time coordinates of the original video, thereby determining a continuous time window. This window is then selected as the optimal candidate visual input in this round of processing.

[0056] In medical dynamic image analysis, such as when analyzing long-term surgical videos to locate critical bleeding steps, the clinical need is often to determine the start and end times of an event, rather than isolated moments. This step can automatically and accurately define the entire time period of the event by finding the continuous time segments with the highest confidence, providing a complete and reliable temporal context for subsequent detailed assessment and report generation.

[0057] 206. Input the best candidate visual input and the user query back into the multimodal large language model to generate and output the response results for the visual perception task request.

[0058] Specifically, the continuous time window corresponding to the optimal subsequence and the user query input are fed into a multimodal large language model to obtain the natural language response generated by the multimodal large language model, where the natural language response is used to describe the content of the target event; the start and end positions of the continuous time window are mapped back to the temporal coordinates of the visual input to obtain the start and end timestamps of the target event content; and the response result for the visual perception task request is generated and output, where the response result includes the natural language response, the start timestamp, and the end timestamp.

[0059] In this implementation, the continuous time window corresponding to the optimal subsequence determined in the preceding steps is first input into the multimodal large language model along with the original user query. The model infers based on the content of this focused time segment and generates a natural language response. The core function of the natural language response is to describe the specific content of the target event that occurred within the time window. Subsequently, the system performs a crucial time information structure transformation, mapping the start and end positions of the continuous time window back to their true time coordinate system based on the temporal metadata of the original visual input. Through this mapping calculation, the precise start and end timestamps of the target event on the original time axis are obtained. Finally, the system integrates the above two parts of information to generate and output a complete response result for the original visual perception task request. The response result is a structured output that contains at least three core elements: the first is the natural language response generated by the model that describes the content of the event; the second is the start timestamp of the event; and the third is the end timestamp of the event.

[0060] In another implementation, after selecting at least one optimal candidate visual input from multiple candidate visual inputs, the method further includes: obtaining contextual visual information adjacent to the optimal candidate visual input based on the temporal or spatial attributes of the optimal candidate visual input; fusing the contextual visual information with the optimal candidate visual input to form an enhanced visual input; and using the enhanced visual input as the optimal candidate visual input for generating a response result.

[0061] Specifically, contextual visual information refers to additional visual data that is directly adjacent to the optimal candidate visual input in the temporal or spatial dimensions. For example, if the optimal candidate is a 2-second time window in a video, the contextual visual information may be video segments 0.5 seconds before and 0.5 seconds after the start of the window; if the optimal candidate is a cropped block in an image, the contextual visual information may be the surrounding image region adjacent to the cropped block. Enhanced visual input is a new visual data unit containing core evidence and context, formed after the fusion operation, used to replace the original optimal candidate visual input for the final response generation step.

[0062] In this implementation, the system first identifies the inherent attributes of the selected optimal candidate visual input, such as its position on a one-dimensional time axis or its coordinates in a two-dimensional image, and automatically locates and acquires adjacent contextual visual information accordingly. Then, the system technically fuses the core optimal candidate with the acquired context to generate an enhanced visual input with richer information. Finally, the system replaces the original visual input used to generate the answer with this enhanced visual input, and the subsequent process remains unchanged. This enhanced input, along with the user query, is input into a multimodal large language model to generate and output the final response. This step compensates for the potential boundary ambiguity or information incompleteness of single evidence by introducing local context. In some complex scenarios, the key evidence that best answers the question may be located at the edge of a candidate clipping block or time window. By actively fusing context, the model is provided with more sufficient judgment criteria, which helps reduce misunderstandings caused by information fragmentation and improves the accuracy and reliability of the final response.

[0063] This application provides an embodiment, taking the time period positioning of a specific action in a rehabilitation training video as an example, to describe in detail the specific implementation process of the present invention. It is assumed that the task is to give a long video of a patient performing rehabilitation training and a text description, such as the patient completing a standard movement from sitting to standing up, and to automatically find the start time and end time of the movement in the video.

[0064] First, candidate inputs are generated by dividing the input rehabilitation training video into a series of continuous and non-overlapping time windows with fixed durations (e.g., 1 second) or fixed frame counts. Each time window in the video is considered an independent candidate visual input. For example, a 5-minute video (300 seconds) will generate 300 candidate visual inputs with 1-second windows.

[0065] Secondly, calculate the uncertainty score. For each time window, construct a binary judgment problem. Based on the user query that the patient has completed the standard action of going from sitting to standing, construct a standard binary problem for the current time window. For example: In this time window, is it described that the patient has completed the standard action of going from sitting to standing? The current time window is then used as visual input, along with the binary question, and fed into a multimodal large language model (MLLM). MLLM outputs a probability distribution for yes or no. In this embodiment, the token entropy formula is used to calculate a scalar value based on the complete probability distribution output by the model, which serves as the uncertainty score for the window. The lower the score, the higher the certainty that the model believes a target action exists in this window. After traversing all time windows, the system obtains a numerical sequence that corresponds one-to-one with the order of the time windows, i.e., a confidence sequence. For example, for an 8-window segment, the possible sequence is: [-0.8, -0.7, 0.1, 0.6, 0.9, 0.85, 0.2, -0.6]. In this example, the values ​​are processed: positive values ​​indicate that the model tends to believe that the action may occur, and the larger the value, the higher the confidence; negative values ​​indicate that the model believes that the action has not occurred or that the uncertainty is extremely high.

[0066] Then, a continuous subsequence is found from the obtained confidence sequence such that the sum of all elements in the subsequence is maximized. This corresponds to finding the continuous time segment in the video with the highest overall model confidence. This is a classic maximum subarray sum problem, which can be solved efficiently in linear time using Kadane's Algorithm. Applying the algorithm to the example sequence [-0.8, -0.7, 0.1, 0.6, 0.9, 0.85, 0.2, -0.6], the algorithm calculates that the continuous subarray with the largest sum is [0.1, 0.6, 0.9, 0.85, 0.2], with a starting index of 3 and an ending index of 7. This optimal subsequence corresponds to the video segment in which the model determines that the target action is most likely to occur continuously. The continuous time window (the 3rd to the 7th window) corresponding to the subsequence is determined as the optimal candidate visual input.

[0067] Finally, the final answer is generated by mapping the start and end indices of the optimal subsequence back to the actual timestamps of the video according to the video segmentation rules (e.g., each window is 1 second long, starting at second 0). The start timestamp is set to second 2, and the end timestamp is set to second 7, corresponding to the end of the 7th window. The obtained optimal candidate visual input (the video segments corresponding to the 3rd to 7th time windows) and the original user query are then input into MLLM again to generate a natural language description as a content summary. For example: The patient completed a continuous movement from sitting to standing during this time period, and the process of getting up was relatively smooth. The system finally outputs a structured response result, including the event description: The patient completed a continuous movement from sitting to standing during this time period, and the process of getting up was relatively smooth, and the location timestamps: the start time is second 2 and the end time is second 7.

[0068] Furthermore, as Figure 1 To specifically implement the method, this application provides a visual perception task processing device, such as... Figure 3 As shown, the device includes: a candidate generation module 301, a candidate evaluation module 302, a candidate screening module 303, and a result output module 304.

[0069] The candidate generation module 301 is used to generate multiple candidate visual inputs corresponding to the received visual perception task request in response to the visual perception task request, wherein the visual perception task request includes visual inputs and user queries. The candidate evaluation module 302 is used to input multiple candidate visual inputs and user query inputs into a preset multimodal large language model, and calculate the uncertainty score used to characterize the multimodal large language model's response to the user query. The candidate selection module 303 is used to select at least one optimal candidate visual input from multiple candidate visual inputs based on the uncertainty score. The result output module 304 is used to input the optimal candidate visual input and the user query back into the multimodal large language model to generate and output the response results for the visual perception task request.

[0070] In specific application scenarios, the candidate generation module 301 is specifically used to crop the image based on multiple preset spatial scales and / or multiple spatial positions when the visual input is an image, generating multiple image cropping blocks as candidate visual inputs; when the visual input is a video, it samples the video based on multiple preset time intervals and / or multiple time points, generating multiple video segments and / or video frames as candidate visual inputs.

[0071] In specific application scenarios, the candidate evaluation module 302 is specifically used to obtain the answer sequence generated by the multimodal large language model in response to user queries, wherein the answer sequence includes multiple tokens; obtain the probability distribution of the tokens on the vocabulary of the multimodal large language model, and calculate the Shannon entropy of the tokens according to the probability distribution; calculate the arithmetic mean of the Shannon entropy of all tokens in the answer sequence, and use the arithmetic mean as the uncertainty score.

[0072] In specific application scenarios, the candidate evaluation module 302 is specifically used to construct a binary judgment question related to the user query based on the candidate visual input; input the candidate visual input and the binary judgment question into a multimodal large language model, obtain the confidence score for the affirmative answer from the output of the multimodal large language model; perform a negative correlation transformation on the confidence score to obtain the transformation value, and use the confidence score or transformation value as the uncertainty score corresponding to the candidate visual input.

[0073] In specific application scenarios, the candidate screening module 303 is specifically used to form a confidence sequence by arranging the uncertainty scores corresponding to the candidate visual inputs according to the order of the candidate visual inputs when multiple candidate visual inputs are arranged in chronological order; by solving the maximum subarray sum algorithm, the continuous optimal subsequence is determined in the confidence sequence, and the continuous time window corresponding to the optimal subsequence is taken as the optimal candidate visual input.

[0074] In specific application scenarios, the result output module 304 is specifically used to input the continuous time window corresponding to the optimal subsequence and the user query into the multimodal large language model, obtain the natural language response generated by the multimodal large language model, wherein the natural language response is used to describe the content of the target event; map the start and end positions of the continuous time window back to the temporal coordinates of the visual input to obtain the start and end timestamps of the target event content; generate and output the response result for the visual perception task request, wherein the response result includes the natural language response, the start timestamp, and the end timestamp.

[0075] In specific application scenarios, such as Figure 4 As shown, the device also includes a candidate enhancement module 305, which is specifically used to obtain the contextual visual information adjacent to the optimal candidate visual input based on the temporal or spatial attributes of the optimal candidate visual input; to fuse the contextual visual information with the optimal candidate visual input to form an enhanced visual input; and to use the enhanced visual input as the optimal candidate visual input for generating the response result.

[0076] It should be noted that other corresponding descriptions of the functional units involved in the visual perception task processing device provided in this embodiment can be found in [reference]. Figure 1 and Figure 2 The corresponding descriptions in [the document] will not be repeated here.

[0077] Based on the above, Figure 1 and Figure 2 Accordingly, this embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described visual perception task processing method.

[0078] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product to be identified can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), including several instructions to enable a computer device (such as a personal computer, server, or network device) to execute the visual perception task processing method of various implementation scenarios of this application.

[0079] Based on the above, Figure 1 and Figure 2 The method shown, and Figure 3 and Figure 4 The illustrated visual perception task processing device embodiment, in order to achieve the above objectives, such as... Figure 5 As shown, this embodiment also provides a physical device for visual perception task processing. This device includes a communication bus, a processor, a memory, and a communication interface. It may also include input / output interfaces and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the visual perception task processing method described in the above embodiment.

[0080] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0081] Those skilled in the art will understand that the structure of the visual perception task processing physical device provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0082] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs to be identified. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the technical solution of this application, firstly, no parameter updates or task-specific training are required for existing multimodal large language models. The model is dynamically guided through a process of generating candidates, calculating scores, selecting the optimal, and responding again. All calculations are completed based on the pre-trained model, reducing application costs and deployment barriers. Secondly, the inherent signals of the multimodal large language model itself, namely the uncertainty score corresponding to its output response, are used as the decision-making basis. The entire process only interacts with a single multimodal large language model and does not integrate any external modules, thereby maintaining the end-to-end simplicity and inherent consistency of the system and reducing the risk of error propagation due to differences between modules. Finally, by actively generating multiple candidate visual inputs and evaluating their uncertainty, the model is essentially guided to perform an adaptive, data-driven focus search on massive input information. This effectively improves the model's ability to complete high-difficulty fine-grained perception tasks without retraining, enhancing its practicality and reliability in professional scenarios. The above method improves the model's fine-grained perception capability while reducing resource consumption by using a unified framework that requires no training, utilizes only the model's inherent uncertainty signals, and maintains end-to-end processing. This enhances its practicality and reliability in challenging fine-grained perception tasks such as small lesion areas in medical imaging.

[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0085] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for processing visual perception tasks, characterized in that, include: In response to a received visual perception task request, multiple candidate visual inputs corresponding to the visual perception task request are generated, wherein the visual perception task request includes visual inputs and user queries; The candidate visual inputs and the user query are respectively input into a preset multimodal large language model, and the uncertainty score used to characterize the multimodal large language model's response to the user query is calculated. Based on the uncertainty score, at least one optimal candidate visual input is selected from the plurality of candidate visual inputs; The optimal candidate visual input and the user query are then input into the multimodal large language model to generate and output a response result for the visual perception task request.

2. The method according to claim 1, characterized in that, The generation of multiple candidate visual inputs corresponding to the visual perception task request includes: When the visual input is an image, the image is cropped based on multiple preset spatial scales and / or multiple spatial positions to generate multiple image cropping blocks as candidate visual inputs; When the visual input is video, the video is sampled based on multiple preset time intervals and / or multiple time points to generate multiple video segments and / or video frames as candidate visual inputs.

3. The method according to claim 1, characterized in that, The calculation, used to characterize the uncertainty score of the multimodal large language model in response to the user query, includes: Obtain the answer sequence generated by the multimodal large language model in response to the user query, wherein the answer sequence includes multiple tokens; Obtain the probability distribution of the token on the vocabulary of the multimodal large language model, and calculate the Shannon entropy of the token based on the probability distribution; The arithmetic mean of the Shannon entropy of all tokens in the answer sequence is calculated, and the arithmetic mean is used as the uncertainty score.

4. The method according to claim 1, characterized in that, The calculation, used to characterize the uncertainty score of the multimodal large language model in response to the user query, includes: Based on the candidate visual input, construct a binary judgment problem related to the user query; The candidate visual input and the binary judgment question are input into the multimodal large language model, and the confidence score for the affirmative answer is obtained from the output of the multimodal large language model. The confidence score is negatively correlated to obtain a transformed value, and the confidence score or the transformed value is used as the uncertainty score corresponding to the candidate visual input.

5. The method according to claim 1, characterized in that, The step of selecting at least one optimal candidate visual input from among the multiple candidate visual inputs based on the uncertainty score includes: When multiple candidate visual inputs are arranged in chronological order, the uncertainty scores corresponding to the candidate visual inputs are arranged in the order of the multiple candidate visual inputs to form a confidence sequence. By solving the maximum subarray sum algorithm, a continuous optimal subsequence is determined in the confidence sequence, and the continuous time window corresponding to the optimal subsequence is used as the optimal candidate visual input.

6. The method according to claim 5, characterized in that, The step of inputting the optimal candidate visual input and the user query back into the multimodal large language model to generate and output a response result for the visual perception task request includes: The continuous time window corresponding to the optimal subsequence and the user query are input into the multimodal large language model to obtain the natural language response generated by the multimodal large language model, wherein the natural language response is used to describe the content of the target event; The start and end positions of the continuous time window are mapped back to the temporal coordinates of the visual input to obtain the start and end timestamps of the target event content. Generate and output a response result for the visual perception task request, wherein the response result includes the natural language response, the start timestamp, and the end timestamp.

7. The method according to claim 1, characterized in that, After selecting at least one optimal candidate visual input from among the plurality of candidate visual inputs, the method further includes: Based on the temporal or spatial attributes of the optimal candidate visual input, obtain the contextual visual information adjacent to the optimal candidate visual input; The contextual visual information is fused with the optimal candidate visual input to form an enhanced visual input, and the enhanced visual input is used as the optimal candidate visual input for generating the response result.

8. A visual perception task processing device, characterized in that, include: A candidate generation module is used to generate multiple candidate visual inputs corresponding to a received visual perception task request, wherein the visual perception task request includes visual inputs and user queries. The candidate evaluation module is used to input multiple candidate visual inputs and the user query into a preset multimodal large language model, and calculate the uncertainty score used to characterize the multimodal large language model's response to the user query. A candidate filtering module is used to filter at least one optimal candidate visual input from a plurality of candidate visual inputs based on the uncertainty score; The result output module is used to input the optimal candidate visual input and the user query back into the multimodal large language model to generate and output the response result for the visual perception task request.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images