A response method, device and equipment of an image analysis task and a storage medium

Abstract

Embodiments of the present application provide a response method and device for an image analysis task, an apparatus and a storage medium, relating to the technical field of artificial intelligence. The response method for the image analysis task comprises: obtaining target feature information of the image analysis task; determining a visual representation set of a target image indicated by the image analysis task; generating initial visual representation information of the image analysis task based on the target feature information and the visual representation set; calling a large language model to process the image analysis task by taking the initial visual representation information as an initial visual context; and after the large language model generates intermediate inference information of a target time step, predicting whether the inference result of the target time step belongs to a risk result based on the intermediate inference information of the target time step, and performing supplementary processing of the visual context if it belongs. Through the present solution, the response efficiency and response accuracy of the image analysis task can be taken into account.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for responding to an image analysis task. Background Technology

[0002] Multimodal large language models have been widely applied to various task processing scenarios; among them, multimodal large language models typically include a visual encoder and a large language model as the language backbone.

[0003] In related technologies, when performing image analysis tasks based on a multimodal large language model, the image indicated by the image analysis task is first encoded into hundreds or even thousands of visual representation units by a visual encoder, and the generated visual representation units can form a visual representation set; then, all visual representation units in the visual representation set are input into the large language model, and the large language model completes the processing of the image analysis task.

[0004] However, inputting a massive number of visual representation units into a large language model would undoubtedly incur enormous computational overhead, resulting in low response efficiency for image analysis tasks. Therefore, how to balance response efficiency and accuracy for image analysis tasks is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, and storage medium for responding to image analysis tasks, so as to achieve both response efficiency and accuracy for image analysis tasks. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a response method for an image analysis task, comprising:

[0007] Obtain target feature information for image analysis tasks;

[0008] Determine the set of visual representations of the target image indicated by the image analysis task;

[0009] Based on the target feature information and the visual representation set, initial visual representation information for the image analysis task is generated; wherein, the initial visual representation information is used to characterize: the visual information of a portion of the visual representation units in the visual representation set;

[0010] Using the initial visual representation information as the initial visual context, a large language model is invoked to process the image analysis task;

[0011] Furthermore, after the large language model generates intermediate inference information for the target time step, it predicts whether the inference result for the target time step belongs to a risky result based on the intermediate inference information for the target time step. If it does, it performs supplementary processing of the visual context.

[0012] Secondly, embodiments of this application provide a response device for an image analysis task, comprising:

[0013] The acquisition module is used to obtain target feature information for image analysis tasks.

[0014] The determination module is used to determine the set of visual representations of the target image indicated by the image analysis task;

[0015] The generation module is used to generate initial visual representation information for the image analysis task based on the target feature information and the visual representation set; wherein, the initial visual representation information is used to characterize: the visual information of a portion of the visual representation units in the visual representation set;

[0016] The calling module is used to call the large language model to process the image analysis task, using the initial visual representation information as the initial visual context.

[0017] The analysis module is used to predict whether the inference result of the target time step is a risky result based on the intermediate inference information generated by the large language model at the target time step. If it is, supplementary processing of the visual context is performed.

[0018] Thirdly, embodiments of this application provide an electronic device, including:

[0019] Memory, used to store computer programs;

[0020] The processor is used to execute the program stored in memory and implement the above-mentioned image analysis task response method.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described response method for the image analysis task.

[0022] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform a response method for any of the image analysis tasks described above.

[0023] Beneficial effects of the embodiments in this application:

[0024] In the image analysis task response method provided in this application embodiment, for the image analysis task, firstly, based on the visual representation set indicated by the image analysis task and the target feature information, initial visual representation information for the image analysis task is generated. This initial visual representation information serves as the initial visual context, and a large language model is invoked to process the image analysis task. Thus, by using a smaller amount of initial visual representation information, the computational overhead of the large language model is reduced. Then, after the large language model generates intermediate inference information for the target time step, based on this intermediate inference information, it is predicted whether the inference result for the target time step is a risky result. If it is, supplementary processing of the visual context is performed. This supplementary processing enhances the richness of the visual context to support subsequent inference processes. Therefore, this solution can balance response efficiency and accuracy for image analysis tasks.

[0025] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these accompanying drawings.

[0027] Figure 1 A flowchart illustrating a response method for an image analysis task provided in an embodiment of this application;

[0028] Figure 2 A flowchart illustrating the implementation of step S103 as provided in this application embodiment;

[0029] Figure 3 A flowchart illustrating the target supplementary information location process provided in the embodiments of this application;

[0030] Figure 4 A schematic diagram illustrating the working principle of the image analysis task response device provided in the embodiments of this application;

[0031] Figure 5 Another flowchart illustrating the response method for an image analysis task provided in an embodiment of this application;

[0032] Figure 6 This is a flowchart illustrating the generation process of the initial compact visual representation provided in the embodiments of this application;

[0033] Figure 7 A schematic diagram of the structure of the response device for the image analysis task provided in the embodiments of this application;

[0034] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0035] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application. In the technical solutions of this application, the operations of obtaining, storing, using, processing, transmitting, providing, and disclosing user personal information are all performed with the user's authorization.

[0036] To facilitate understanding of the technical solutions provided in this application, the technical terms involved in this application will be introduced.

[0037] Multimodal Large Language Model (MLLM): A large-scale language model capable of processing multiple modal inputs such as text and images simultaneously. It typically includes a visual encoder, a modality alignment module, and a large language model, which can generate text responses based on image content and text instructions.

[0038] Visual Token: Each feature vector in the feature vector sequence obtained after the visual encoder encodes the input image (i.e., each visual token is a feature vector in the feature vector sequence). Each visual token corresponds to a local region (i.e., a sub-region) in the image and carries the visual semantic information of that local region.

[0039] Visual Cache: A storage structure used to store the complete set of visual representations obtained after encoding by the visual encoder, so that visual representation units can be retrieved from it as needed during inference without re-encoding the visual representations.

[0040] Answer Risk Indicator: In the decoding process of a multimodal large language model, this numerical indicator is calculated based on intermediate inference information (such as output probability distribution) at the current time step. It is used to quantify the uncertainty or unreliability of answer generation (such as the inference result at the current time step) supported by current visual evidence (i.e., the current visual context). For example, it is calculated based on sub-indices such as output entropy, candidate probability interval, candidate dispersion, output confidence, and risk prior weights for task requirement types. Higher values ​​indicate greater instability and higher risk in answer generation. Here, "answer" broadly refers to the output content generated by the multimodal large language model in response to input text instructions and image data, including but not limited to answers to questions, image descriptions, and instruction execution results. For a detailed explanation of output entropy, candidate probability interval, candidate dispersion, output confidence, and risk prior weights for task requirement types, please refer to the examples below.

[0041] Visual Evidence Supplementation: In the answer generation process (i.e., the processing of image analysis tasks by a large language model), when the answer risk index is detected to exceed a predetermined risk threshold, visual representation units related to the image analysis task (e.g., the question for the image) and the generation state are extracted (e.g., extracted from the visual cache) to construct supplementary information and injected into the current multimodal context (e.g., the visual context).

[0042] The generated state can be characterized by at least one of the following: the sequence of inference results currently generated by the large language model (or it can be generated as partial answer content), the target task requirement type of the image analysis task, and answer risk indicators.

[0043] Initial Compact Visual Representation: A smaller set of visual representations obtained from the complete set of visual representations through compression (such as summarization or subset selection) is used to replace the complete set of visual representations as input to the large language model in the initial stage of inference, thereby reducing the initial inference overhead.

[0044] Region Type Metadata: This can be simply referred to as metadata. It refers to the semantic attribute annotation information of each region of an image. For example, it includes category labels such as subject region, background region, region with rich detail, salient region, and region with dense structure. In other words, candidate region types can include subject region, background region, region with rich detail, salient region, and region with dense structure. The identification methods of various region types are described in the following examples.

[0045] Target Entity Category: The entity category of the entity objects of interest obtained from the image analysis task. For example, the entity category of the main entity objects of interest in the current problem obtained from the text input of the current problem, such as people, vehicles, pets, etc.

[0046] Task Demand Type: Classification of task nature obtained from image analysis task parsing. For example, classification of the task nature of the current problem obtained from parsing the text input content of the current problem, such as attribute-type requirements, counting-type requirements, relation-type requirements, subject recognition-type requirements, scene semantic requirements, local detail recognition-type requirements, etc.

[0047] To better understand the technical solution provided in this application, the relevant technologies are introduced below:

[0048] In related technologies, when processing images using multimodal large language models, visual encoders typically encode an image into hundreds or even thousands of visual representation units. Inputting all of these visual representation units into a large language model results in enormous computational overhead and key-value cache usage.

[0049] To address this issue, various visual representation unit compression schemes have been proposed, such as a scheme that performs a one-time static compression of the visual representation unit before inference begins.

[0050] The working process of this one-time static compression scheme is as follows: after the image is input into the visual encoder to obtain the visual representation set, the subset of visual representation units that need to be retained is determined once before the inference begins through attention score sorting, importance evaluation or merging operations, the remaining visual representation units are discarded, and then the retained subset of visual representation units is fed into the large language model together with the image analysis task (such as text input) for inference.

[0051] The aforementioned technologies have the following technical problems:

[0052] (1) The compression decision is disconnected from the answer generation process: The existing schemes mentioned above complete the retention decision of visual representation units before the reasoning begins. This retention decision cannot perceive the actual needs in the subsequent answer generation process. When the large language model encounters key positions that require fine-grained visual information during the answer generation process, the discarded visual representation units cannot be recovered, resulting in a decrease in answer quality.

[0053] (2) Static compression strategies cannot adapt to problems of varying difficulty: For simple problems (such as scene description), a small number of visual representation units are sufficient to support answer generation; for complex problems (such as fine-grained attribute recognition, text recognition, and target counting), more local detail information is required. Existing one-time compression schemes use a fixed compression ratio or strategy, which cannot dynamically adjust the amount of visual information according to the difficulty of the problem and the answer generation status.

[0054] (3) The discarded visual information is irrecoverable: The existing scheme directly discards the unselected visual representation units after compression. Once the compression decision is wrong or the visual information that needs to be discarded in the subsequent generation process is lost, it cannot be remedied, resulting in performance loss on fine-grained visual tasks.

[0055] (4) The compression strategy does not distinguish between target entities and background: The existing solutions mentioned above adopt a uniform compression strategy for all visual regions, without differentiating different regions according to the target entity category and task requirements. For example, when the question asks for the color of a person's clothing in an image, the visual representation units of the region where the person is located should be retained first. However, the existing solutions mentioned above cannot perceive this requirement and may treat the visual representation units of the key target entity-related regions the same as the background regions or even discard them first.

[0056] The image analysis task response method provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0057] like Figure 1 As shown in the embodiments of this application, a response method for an image analysis task may include:

[0058] S101, Obtain target feature information for the image analysis task;

[0059] S102, determine the set of visual representations of the target image indicated by the image analysis task;

[0060] S103, Based on the target feature information and the visual representation set, generate initial visual representation information for the image analysis task; wherein, the initial visual representation information is used to characterize: the visual information of a portion of the visual representation units in the visual representation set;

[0061] S104, using the initial visual representation information as the initial visual context, the large language model is invoked to process the image analysis task;

[0062] S105, after the large language model generates intermediate inference information for the target time step, based on the intermediate inference information for the target time step, predict whether the inference result for the target time step belongs to a risky result. If it does, perform supplementary processing of the visual context.

[0063] The response method for this image analysis task can be applied to electronic devices, which can be terminal devices or server devices. Furthermore, the executing entity of this image analysis task response method can be an image analysis task response device. This response device responds to the task based on a multimodal large language model. For example, the product form of the response device includes, but is not limited to, intelligent agent form, functional plug-in form, independent client program form, web cloud service form, system built-in component form, etc.

[0064] There are various ways to acquire image analysis tasks. For example, image analysis tasks can be received from users through a human-computer interaction interface. In addition, image analysis tasks can also be acquired through external interface calls, local / cloud file import, real-time acquisition by vision devices, automatic system triggering, message queue push, etc. This application does not limit the specific acquisition method.

[0065] Furthermore, the image analysis task can be a text-based task, such as a text-based question, instruction, or prompt, but it is not limited to these. Moreover, the image analysis task can be a task given in any dialogue process (also called a dialogue turn) within a multi-turn dialogue scenario focusing on the same target image, or it can be a task given in a single-turn dialogue scenario focusing on the target image; both are reasonable. It is understood that in a multi-turn dialogue scenario, only the target image needs to be input in the first turn of the dialogue; subsequent dialogue processes do not require repeated image input but only the image analysis task. In a single-turn dialogue scenario, the image analysis task and the target image can be input simultaneously so that the response device can obtain both the target image and the image analysis task. Both the aforementioned multi-turn and single-turn dialogue scenarios belong to the interaction scenarios between the user and the response device (i.e., the multimodal large language model), and the distinction between the two is based on the number of interaction turns.

[0066] After obtaining the image analysis task, the image analysis task can be parsed (i.e., the task content of the image analysis task can be parsed) to obtain target feature information. This target feature information is used to describe the image analysis task from a predetermined dimension, which can be a dimension such as the analysis object or the task requirement type. For example, the target feature information may include the target entity category of the image analysis task, that is, the entity category of the entity object to be analyzed by the image analysis task; or, the target feature information may include the target task requirement type of the image analysis task; or, the target feature information may include both the target entity category and the target task requirement type of the image analysis task. It should be noted that, for example, the target entity category and the target task requirement type can be obtained through rule parsing, lightweight text classification models, keyword mapping methods, or combinations thereof. This application does not limit the specific analysis method.

[0067] Furthermore, after obtaining the image analysis task, the visual representation set of the target image indicated by the image analysis task can be determined for use in subsequent processing. The visual representation set of the target image is a set constructed based on multiple visual representation units encoded by the visual encoder, and this visual representation set may contain all visual representation units of the target image. Moreover, this visual representation set can be cached, thereby enabling its application in the visual verification process of the same round of dialogue and / or in multiple rounds of dialogue, such as through the aforementioned visual caching.

[0068] In one implementation, determining the set of visual representations of the target image indicated by the image analysis task may include:

[0069] If the visual representation set of the target image indicated by the image analysis task is not cached, the visual representation set is constructed based on the target image; if the visual representation set is cached, the cached visual representation set is obtained.

[0070] Furthermore, after constructing the visual representation set, the visual representation set is cached;

[0071] Specifically, after obtaining the image analysis task, if the visual representation set of the target image is not currently cached, a visual encoder can be invoked to encode the target image to obtain multiple visual representation units, thereby constructing a visual representation set. If the visual representation set of the target image is already cached, it can be directly determined without invoking the visual encoder. In other words, determining the visual representation set of the target image indicated by the image analysis task can include: if the visual representation set of the target image is cached, obtaining the cached visual representation set of the target image; if the visual representation set of the target image is not cached, invoking the visual encoder (also called the visual encoding module) to encode multiple visual representation units, constructing the visual representation set of the target image based on the generated visual representation units, and caching the constructed visual representation set.

[0072] Among them, the set of visual representations of the target image can be obtained through To characterize, ,in, Indicates the first The visual representation unit (also known as the first) (visual features corresponding to each visual representation unit) This represents the total number of visual representation units. Furthermore, it should be noted that while constructing the visual cache (also known as establishing a visual cache, i.e., writing the visual representation set and related content into the storage structure), visual index information can also be established to describe the correspondence between each visual representation unit and the original image spatial region (i.e., the sub-region / local region mentioned above), resolution level (also known as the image multi-scale level), region category (such as the metadata of the region unit to which it belongs) or cache address, so as to locate the target supplementary information required for visual supplementation (i.e., locate the visual representation unit used for supplementary processing; the target supplementary information can also be called supplementary visual evidence).

[0073] After obtaining the target feature information and visual representation set, in order to reduce the computational overhead of the large language model, this embodiment does not input all visual representation units in the visual representation set as visual context into the large language model. Instead, based on the target feature information and visual representation set, initial visual representation information (also called an initial compact visual representation set) for the image analysis task can be generated. Then, using this initial visual representation information as the initial visual context, the large language model is invoked to process the image analysis task. This processing is a multi-time-step process. It is understood that when invoking the large language model, the task content of the image analysis task can also be input into the large language model.

[0074] It is understood that, through step S103, the visual representation set can be used as a basis. Generate initial visual representation information And satisfy: ,in, This represents visual compression, summary generation, region aggregation, or subset selection operations. The initial visual representation information... The size is smaller than the visual representation set. The scale, that is: Therefore, since the initial visual representation information is used as visual input in the initial stage of the large language model, the computational cost and caching overhead of the initial inference can be reduced.

[0075] In addition, in this embodiment, taking the image analysis task as text input T as an example, through step S104, the text input belonging to the image analysis task can be... With the initial visual representation information Input the large language model, execute the answer generation process (i.e., the task processing process), and obtain the answer. Output probability distribution at each time step:

[0076] ;

[0077] in, It can represent the processing function for task response processing of large language models. Indicates the first The output token to be generated at each time step; Indicates the first The output sequence generated before each time step; Indicates the visual context corresponding to the current time step, initial time command .

[0078] There are multiple ways to generate the initial visual representation information for the image analysis task based on the target feature information and the visual representation set.

[0079] For example, in one implementation, for each visual representation unit in the visual representation set, the similarity between the vector value of the temporal representation unit and the target feature information can be calculated. Then, visual representation units with similarity higher than a predetermined selection threshold are selected, and the selected visual representation units or the information after compression of the selected visual representation units are used as the initial visual representation information.

[0080] For example, in another implementation, the visual representation set can correspond to a region set containing multiple region units. Each region unit contains visual representation units in the visual representation set whose corresponding sub-regions belong to the same local image region. Each region unit corresponds to a local image region. For each region unit, the similarity between the vector value of the visual representation unit in the region unit and the vector value of the target feature information can be calculated. Region units with a similarity higher than a predetermined selection threshold are selected. The visual representation units in the selected region units, or the information after compression of the visual representation units in the selected region units, are used as the initial visual representation information. Alternatively, label information (such as entity category labels) can be set for each region unit. The region unit is selected based on whether the label information matches the target feature information. The initial visual representation information is determined based on the selected region units.

[0081] For example, in other implementations, each region unit is assigned an entity category label, which characterizes the entity category of the local image region to which the visual representation unit within that region unit belongs. Each region unit also has metadata, which characterizes the region type (i.e., one or more of the candidate region types) of the local image region to which the visual representation unit within that region unit belongs. The initial visual representation information can be obtained through any of the following methods or combinations thereof: prioritizing the retention of visual representation units within region units whose entity category labels match the target entity category; selecting representative visual representation units from the visual representation set (e.g., visual representation units within region units belonging to salient regions, subject regions, etc.); generating multiple summary visual representation units based on the complete visual representation; compressing the complete visual representation based on spatial regions, object regions, text regions, or resolution levels; and determining the initial retention scale based on the problem type, budget constraints, or lightweight risk prediction results.

[0082] It should be emphasized that other implementations of generating the initial visual representation information for the image analysis task based on the target feature information and the visual representation set will be described in the following embodiments.

[0083] In addition, to ensure the accuracy of the image analysis task response, the processing of the large language model can be monitored. After the large language model generates intermediate inference information for the target time step, it can predict whether the inference result of the target time step is a risky result based on the intermediate inference information. If it is, supplementary processing of the visual context is performed to supplement the visual evidence and support the subsequent inference process. That is, after predicting that the inference result of the target time step is a risky result, the visual evidence supplementation process can be triggered, that is, the supplementary processing of the visual context is triggered.

[0084] There are several ways to determine the target time step. For example, in one implementation, each time step in the large language model's inference process can be used as a target time step. Correspondingly, after the large language model generates intermediate inference information for each time step, it predicts whether the inference result for that time step is a risky result based on that intermediate inference information. For example, in another implementation, at least one predetermined time step in the large language model's inference process can be used as a target time step. Correspondingly, after the large language model generates intermediate inference information for each predetermined time step, it predicts whether the inference result for that predetermined time step is a risky result based on that intermediate inference information. The method for determining the at least one predetermined time step can include, but is not limited to, selecting it randomly, or selecting it at predetermined step intervals, etc.

[0085] The intermediate inference information of the target time step can be: the output probability distribution of the target time step, which is the output probability of each candidate word (candidate token); the inference result of the target time step can be: the output word (output token) of the target time step.

[0086] To facilitate understanding of the intermediate inference information and results at the target time step, the working principle of a multimodal large language model is introduced below, using an image analysis task as the text input T as an example:

[0087] The multimodal large language model uses an autoregressive decoding method to generate the output sequence step by step. Each time step contains the following two sub-processes executed sequentially:

[0088] (1) Probability distribution calculation sub-process: For a task, in the first... At each time step, the large language model is based on the text input for this task. Current visual context and the generated output sequence (That is, the output sequence or inference result sequence generated based on the output tokens of historical time steps) Perform a forward computation to obtain the output probability distribution of the current time step (i.e., the t-th time step) on the vocabulary. The output probability distribution provides the probability value of each candidate token in the vocabulary as the current output token. At this point, the output token for the current time step is not yet determined. Indicates the first The output token to be generated at each time step.

[0089] (2) Token selection sub-process: According to the preset decoding strategy, the output token for the current time step is selected from the candidate tokens corresponding to the output probability distribution (i.e., The generated output sequence is then appended to it as input for the next time step. The decoding strategies include, but are not limited to, greedy decoding (selecting the candidate token with the highest probability), sampling decoding (sampling according to probability distribution), top-k sampling, top-p sampling, or beam search.

[0090] Therefore, it can be seen that the large language model first obtains the output probability distribution covering the entire vocabulary at each time step. The output token of the current time step is selected from the candidate tokens through the token selection subprocess after the probability distribution is calculated, rather than directly generating a unique final output token after forward calculation.

[0091] The technical solution provided in this embodiment utilizes the temporal interval between the probability distribution calculation sub-process and the token selection sub-process: after the probability distribution calculation is completed and before the current output token is selected, it is possible to predict whether the reasoning result of the target time step belongs to a risky result based on the output probability distribution (such as calculating the answer risk index and making a risk trigger judgment); thus, in the case of a risky result, supplementary processing of the visual context is performed.

[0092] It is understandable that there can be multiple ways to implement supplementary processing of visual context. For example, in one implementation, visual representation units that meet filtering criteria are selected from the set of visual representations. The selected visual representation units, or information compressed from the selected visual representation units, are added to the current visual context as target supplementary information. The filtering criteria may include: the context belonging to the visual representation unit indicated by the current visual context (e.g., adjacent image regions), or the region unit belonging to a specified region type (e.g., salient region, detailed region, subject region, etc.). For clarity and layout, the implementation methods for performing supplementary processing of visual context will be exemplarily described in other embodiments below.

[0093] It is important to emphasize that after performing supplementary visual context processing, an updated visual context is obtained. Furthermore, based on the updated visual context, the decoding process of the large language model can continue to generate subsequent answer tokens (lexical units), i.e., generate subsequent partial answer content, until a preset stopping condition is met. For example, the stopping condition includes, but is not limited to: generating a termination token; reaching the maximum output length; the answer risk decreasing below a threshold and completing the current answer generation; reaching the maximum number of supplementary verification attempts (i.e., the maximum number of supplementary processing attempts); and reaching the preset visual supplementary verification budget (i.e., the lexical budget set for visual supplementary verification in the entire response process).

[0094] In the image analysis task response method provided in this application embodiment, for the image analysis task, firstly, based on the visual representation set indicated by the image analysis task and the target feature information, initial visual representation information for the image analysis task is generated. This initial visual representation information serves as the initial visual context, and a large language model is invoked to process the image analysis task. Thus, by using a smaller amount of initial visual representation information, the computational overhead of the large language model is reduced. Then, after the large language model generates intermediate inference information for the target time step, based on this intermediate inference information, it is predicted whether the inference result for the target time step is a risky result. If it is, supplementary processing of the visual context is performed. This supplementary processing enhances the richness of the visual context to support subsequent inference processes. Therefore, this solution can balance response efficiency and accuracy for image analysis tasks.

[0095] In one possible embodiment, such as Figure 2 As shown, generating the initial visual representation information for the image analysis task based on the target feature information and the visual representation set may include:

[0096] S201, determine the region set corresponding to the visual representation set, which contains multiple region units;

[0097] Each region unit contains a visual representation unit from the visual representation set whose corresponding sub-region belongs to the same local region of the image;

[0098] S202, Based on the target feature information, select region units from the region set to obtain target region units;

[0099] S203, Based on the visual representation units in the obtained target region units, construct the initial visual representation information for the image analysis task.

[0100] In this embodiment, the visual representation set of the target image corresponds to a region set, and the region set contains multiple region units.

[0101] For example, the generation of the region set may include: based on the mapping relationship between each visual representation unit and the sub-region, dividing the visual representation units in the visual representation set into multiple region units according to their spatial location to construct the region set. Furthermore, each region unit may be provided with metadata (i.e., the region type metadata mentioned above), and the metadata represents: the region type of the local image region to which the visual representation unit within that region unit belongs (i.e., one or more types among the candidate region types).

[0102] Among them, the region set can be obtained through To characterize, Where M is the total number of regional units in the region set. Let j be the j-th region unit; and, The corresponding visual representation unit subset (i.e. The subset formed by the various visual representation units contained within it can be accessed through... To characterize, , For the first A visual representation unit. It can be understood that, exemplarily, a set of visual representations... It can be stored in the visual cache and represented as: ,in, Used to represent visual buffers; additionally, it can record the set of indices of visual representation units corresponding to each region unit:

[0103] ;

[0104] in, For regional units The corresponding set of indices Used to represent region units In The index information. Therefore, the above index set can be used to extract supplementary information about the target (i.e., supplementary visual evidence) after an answer risk is triggered.

[0105] In addition, the candidate region types can include, but are not limited to: subject region, background region, region rich in detail, salient region, densely structured region, etc. Among them, region rich in detail can be determined based on the degree of variation of visual features within the region; salient region can be determined based on the degree of correlation between the feature information of the region unit and the global feature information (also known as global feature representation or global visual representation), the degree of difference between the region and its neighboring regions, or a combination thereof.

[0106] Optionally, the method for determining the metadata set for each regional unit may include:

[0107] For each region unit, feature information (also known as region feature representation) of that region unit is generated based on the visual representation units within that region unit.

[0108] Based on the feature information of each regional unit, metadata for each regional unit is generated; among them, the metadata of the j-th regional unit can be obtained through... Characterize it.

[0109] For example, the feature information of each region unit can be characterized by the following formula:

[0110] ;

[0111] in, For the feature information of the j-th region unit, This represents a region aggregation function. It is the subset of visual representation units corresponding to the j-th region unit (i.e., the visual representation units contained in the j-th region unit).

[0112] To facilitate understanding of how metadata for each regional unit is generated based on its feature information, the following description, in conjunction with other embodiments, will illustrate this approach.

[0113] In addition, each region unit can be assigned an entity category label to represent the entity category of the local image region to which the visual representation unit within that region unit belongs. For example, based on the feature information of each region unit, the entity category label of each region unit can be determined and represented by the following formula:

[0114] ;

[0115] in, For the entity category label of the j-th region unit, This represents the entity category determination function (also known as the entity relevance determination function).

[0116] After determining the set of regions, the target feature information is used as the filtering criterion to select region units from the set of regions to obtain target region units; then, based on the visual representation units in the obtained target region units, the initial visual representation information of the image analysis task is constructed.

[0117] There are various ways to construct the initial visual representation information for the image analysis task based on the visual representation units in the obtained target region units. For example, in one implementation, the visual representation units in the obtained target region units can be directly used as the initial visual representation information for the image analysis task; alternatively, the visual representation units in the obtained target region units can be compressed according to a predetermined compression method, and the compressed information can be used as the initial visual representation information. Other implementation methods for constructing the initial visual representation information for the image analysis task are described in the embodiments below.

[0118] Furthermore, since the content included in the target feature information is different, the method of selecting region units from the region set can also be different. For example, if the target feature information includes the target task requirement type of the image analysis task, the predetermined metadata corresponding to the target task requirement type can be determined based on the correspondence between each task requirement type and the predetermined metadata. Region units with the determined predetermined metadata can be selected from the region set. The predetermined metadata corresponding to each task requirement type is used to characterize the region type to be used under that task requirement type.

[0119] The implementation method for selecting region units from the region set based on the target feature information is exemplarily described in the following embodiments.

[0120] In this embodiment, after obtaining the target feature information and the region set, region units can be selected from the region set based on the target feature information to obtain target region units; based on the visual representation units in the obtained target region units, the initial visual representation information of the image analysis task is constructed; thus, through this embodiment, the initial visual representation information of the image analysis task can be constructed quickly and effectively.

[0121] In one possible embodiment, to facilitate understanding of how metadata for each region unit is generated based on its feature information, the following example illustrates how to determine whether any region unit has metadata representing any candidate region type, using candidate region types including detailed regions, salient regions, structurally dense regions, main regions, and background regions as examples:

[0122] The methods for determining whether any regional unit has metadata representing a region with rich details include:

[0123] According to the first formula, calculate the degree of visual feature change within the region unit. If the degree of visual feature change is greater than the preset change threshold, set metadata representing a region with rich details for the region unit.

[0124] For example, the first formula can be as follows:

[0125] ;

[0126] in, The degree of visual feature change within the j-th region unit. For the subset of visual representation units corresponding to the j-th region unit, For the first Each visual representation unit.

[0127] The methods for determining whether any regional unit has metadata representing a significant region may include:

[0128] Determine the global feature information of the target image; wherein, the global feature information can be calculated using the second formula:

[0129] According to the third formula, the similarity between the feature information of the region unit and the global feature information is calculated; if the similarity is less than the predetermined similarity threshold, metadata representing a significant region is set for the region unit.

[0130] For example, the second formula can be as follows:

[0131] ;

[0132] in, The visual representation is the global feature information of the target image, and N is the total number of visual representation units in the visual representation set. For the first Each visual representation unit.

[0133] For example, the third formula can be as follows:

[0134] ;

[0135] in, The similarity between the j-th regional unit and the global feature information is denoted as .

[0136] The methods for determining whether any regional unit has metadata representing a densely structured region may include:

[0137] According to the fourth formula, the degree of feature difference between the region unit and the neighboring region units is calculated; if the degree of feature difference is greater than the preset difference threshold, metadata representing the dense structure region is set for the region unit.

[0138] For example, the fourth formula can be as follows:

[0139] ;

[0140] in, The degree of feature difference between the j-th regional unit and its neighboring regional units. Let j be the target set of the neighboring regional units of the j-th regional unit. This refers to the k-th region unit in the target set.

[0141] The methods for determining whether any regional unit has metadata representing the main or background region can include:

[0142] If the degree of change of visual features within the region unit is not greater than a preset change threshold, the degree of feature difference between the region unit and neighboring region units is not greater than a preset difference threshold, and the degree of similarity between the feature information of the region unit and the global feature information is not less than a predetermined similarity threshold, then metadata representing the background region is set for the region unit; otherwise, metadata representing the subject region is set for the region unit.

[0143] In one possible embodiment, the target feature information includes: the target entity category indicated by the image analysis task;

[0144] The step of selecting target region units from the region set based on the target feature information can include:

[0145] Step A1: Determine the entity category label for each region unit; wherein, the entity category label for each region unit is used to characterize: the entity category of the local image region to which the visual representation unit within that region unit belongs;

[0146] Step A2: Based on the entity category label of each region unit, select the target region unit whose entity category label matches the target entity category from the region set.

[0147] In this embodiment, the target feature information includes the target entity category indicated by the image analysis task. Based on the target entity category, region units are selected to obtain target region units, that is, region units that are related to the image analysis task.

[0148] It should be noted that when the target entity category is unknown, or the set of target entity related regions is empty (i.e., no target region unit matching the entity category label and the target entity category is selected), for example, the union of the significant region and the main region is used as the set of backup target regions, that is, as the set of selected target region units.

[0149] To better understand how the target region unit is selected, please refer to the following formula:

[0150] ;

[0151] in, The set of regions related to the target entity (a set of target region units whose entity category labels match the target entity category); For the entity category label of the j-th region unit, For the target entity category;

[0152] From this formula, we can see that when... When, determine the first Each region unit is a target entity-related region, that is, a target region unit whose entity category label matches the target entity category; thus, a set of target entity-related regions is obtained.

[0153] In this embodiment, based on the entity category labels of each region unit and the target entity labels of the image analysis task, the relevant regions of the target entity can be selected from the region set. That is, the relevant regions of the entities of interest to the image analysis task are selected first. Thus, it can be ensured that the initial visual representation information has a high correlation with the image analysis task, providing a basis for improving response accuracy.

[0154] In one possible embodiment, each region unit is provided with metadata, and the metadata represents the region type of the local image region to which the visual representation unit within that region unit belongs;

[0155] Before constructing the initial visual representation information for the image analysis task based on the visual representation units in the obtained target region units, the method further includes:

[0156] Step B1: Based on the metadata of each region unit, select region units with specified metadata from the region set to obtain the global context region;

[0157] Wherein, the region type represented by the specified metadata is: a region type that can represent the scene semantics of the target image;

[0158] The initial visual representation information for the image analysis task, constructed based on the visual representation units in the obtained target region units, may include:

[0159] Step B2: Based on the visual representation units in the obtained target region units and the visual representation units of the global context region, construct the initial visual representation information for the image analysis task.

[0160] In this embodiment, in order to avoid the initial visual input (i.e., the initial visual representation information) being overly concentrated in the target entity-related region (i.e., the target region unit mentioned above) and losing global information, before constructing the initial visual representation information of the image analysis task, a region unit with specified metadata can be selected from the region set based on the metadata of each region unit to obtain the global context region, thereby preserving the overall scene semantics of the image for the large language model.

[0161] It is understood that the global context region is the region other than the target entity-related region in the visual representation set (i.e., other region units other than the target region unit). For example, the region type represented by the specified metadata may include, but is not limited to: background region, detailed region, etc.

[0162] There are multiple ways to construct the initial visual representation information for the image analysis task based on the visual representation units in the obtained target region unit and the visual representation units in the global context region. For example, the visual representation units in the obtained target region unit and the visual representation units in the global context region can be directly combined to obtain the initial visual representation information for the image analysis task. For clarity and layout, other embodiments are described below.

[0163] In this embodiment, while obtaining the target region unit, the global context region is also obtained. Thus, based on the target region unit and the global context region, initial visual representation information is constructed. This avoids the initial visual representation information from being overly concentrated in the target entity-related region and losing global information, thereby providing a foundation for further improving the response accuracy of image analysis tasks.

[0164] In one possible embodiment, constructing the initial visual representation information for the image analysis task based on the visual representation units in the obtained target region unit and the visual representation units of the global context region may include:

[0165] Step B21: Compress the visual representation units in the obtained target region unit and the visual representation units in the global context region to obtain initial visual representation information.

[0166] The compression process may include:

[0167] Filtering operations for region units, and / or summarizing operations for region units;

[0168] The filtering operation for region units includes: filtering operations performed according to a predetermined region unit retention strategy and the target task requirement type of the image analysis task. The predetermined region unit retention strategy includes: the mapping relationship between each task requirement type and the region retention rule.

[0169] The summary generation operation for the region unit includes at least one of the following operations:

[0170] For any region unit, select a portion of the visual representation units within that region unit;

[0171] For any region unit, the visual representation units within that region unit are pooled.

[0172] For any given region unit, the visual representation units within that region unit are mapped to a predetermined number of summary contents.

[0173] To further reduce computational overhead, the visual representation units in the obtained target region unit and the visual representation units in the global context region can be compressed to obtain initial visual representation information. The number of tokens in this initial visual representation information is less than the total number of tokens in the obtained target region unit and the visual representation units in the global context region.

[0174] For filtering operations related to region units, this method is used to filter target region units and / or global context regions. When filtering, the target task requirement type of the image analysis task can be used as the analysis benchmark, combined with a predetermined region unit retention strategy. The predetermined region unit retention strategy may include: a mapping relationship between each task requirement type and a region retention rule. In this mapping relationship, the region retention rule mapped to each task requirement type is the region unit that should be preferentially retained under that task requirement type (i.e., the region unit with the highest retention priority).

[0175] For example, taking the task requirement types including attribute requirements, local detail recognition requirements, counting requirements, relationship requirements, subject recognition requirements, and scene semantic requirements as examples, the mapping relationship between each task requirement type and the region preservation rule can be as follows:

[0176] Attribute-based requirements or local detail recognition requirements are mapped as follows: prioritize preserving areas with rich details and main body areas;

[0177] The requirements mapping for counting types includes: prioritizing the preservation of densely structured regions;

[0178] Relationship-based requirement mappings include: prioritizing the retention of the main area and the spatial relationship-sensitive areas of multiple entities;

[0179] Subject recognition requirements include: prioritizing the preservation of the main subject area and areas rich in detail;

[0180] The scene semantic class requirement mapping includes: prioritizing the retention of the main area and a predetermined number of global context areas.

[0181] The above example of the mapping relationship between various task requirement types and area retention rules can be understood as setting an area retention priority for each task requirement type, as detailed below:

[0182] When the task requirement is attribute-based or local detail recognition-based, priority should be given to retaining the summary visual representation of the detailed areas and the main body area (the so-called summary visual representation can be understood as the representation information obtained after compression).

[0183] When the task requirement type is a counting requirement, priority is given to retaining the summary visual representation of densely structured areas to ensure that the area where each instance (i.e., entity instance) is located is covered by visual representation units;

[0184] When the task requirement type is a relationship-based requirement, priority should be given to retaining the summary visual representation of the main area and the spatial relationship-sensitive area containing multiple entities;

[0185] When the task requirement is subject recognition, priority should be given to retaining the summary visual representation of the subject area and the detailed area.

[0186] When the task requirement type is a scene semantic requirement, priority should be given to retaining the main area and a small amount of global context summary visual representation.

[0187] Furthermore, regarding the aforementioned summary generation operation for region units, the pooling process may include, but is not limited to, average pooling; and the selected visual representation units can be considered representative visual representation units, such as visual representation units within region units belonging to the main body region or salient region. In other words, the generation method of the summary visual representation units (i.e., the summary generation operation for region units) includes at least one of the following: selecting a portion of representative visual representation units from the visual representation units within the region unit; performing pooling operations such as average pooling on the visual representation units within the region unit to obtain region-level summary visual representation units; and mapping multiple visual representation units within the region unit into a smaller number of summary visual representation units using a lightweight compression module.

[0188] It is understood that the initial visual representation information (also known as the initial compact visual representation set) consists of at least two parts: a summary visual representation of the target entity’s relevant region (i.e., the information obtained by compressing the target region unit as described above) and a summary visual representation of the global context (i.e., the information obtained by compressing the global context region as described above).

[0189] The relationship between the initial visual representation information and these two components can be characterized by the following formula:

[0190] ;

[0191] in, This serves as the initial visual representation of information. A summary visual representation of the relevant regions of the target entity; A summary visual representation of the global context.

[0192] In this embodiment, the visual representation units in the obtained target region unit and the visual representation units in the global context region are compressed to obtain initial visual representation information, thereby further reducing computational overhead and improving response rate. Furthermore, the above compression process can ensure the relevance of the initial visual representation information to the image analysis task, that is, to ensure that the initial visual representation information has high adaptability to the image analysis task, so as to ensure response accuracy.

[0193] In one possible embodiment, the compression process of the visual representation units in the obtained target region unit and the visual representation units in the global context region to obtain initial visual representation information may include:

[0194] Step B211: According to the first lexical budget and the second lexical budget, the visual representation units in the obtained target region units and the visual representation units in the global context region are compressed respectively to obtain the initial visual representation information.

[0195] Wherein, the first word element budget and the second word element budget are obtained by allocating the set total word element budget according to the first allocation rule and / or the second allocation rule;

[0196] The first allocation rule is: the lexical budget for the target region unit is higher than the lexical budget for the global context region;

[0197] The second allocation rule is as follows: when the target task requirement type of the image analysis task is the first type of requirement, the lexical budget for the target region unit is higher than the lexical budget for the global context region; when the target task requirement type of the image analysis task is the second type of requirement, the lexical budget for the global context region is higher than the lexical budget for the target region unit.

[0198] The first type of requirement is one that focuses on local details, while the second type of requirement is one that focuses on global semantics.

[0199] To avoid excessive computational overhead, the initial visual representation information can be set with a total lexical budget (which can be called budget constraint B, serving as the initial visual representation information). The total number of visual representation units is capped; based on this total token budget, token budgets can be set for target region units and global context regions respectively.

[0200] Therefore, in the process of constructing the initial visual representation information for the image analysis task, a first token budget for the target region unit and a second token budget for the global context region can be determined first. Then, according to the first token budget and the second token budget, the visual representation units in the obtained target region unit and the visual representation units in the global context region are compressed to obtain the initial visual representation information. That is, the visual representation units in the obtained target region unit are compressed according to the first token budget, and the obtained global context region is compressed according to the second token budget, so that the number of tokens (token count) of the information about the target region unit in the initial visual representation information matches the first token budget, and the number of tokens of the information about the global context region matches the second token budget.

[0201] For the first allocation rule:

[0202] For target entity-related regions (i.e., target region units), a relatively higher retention priority can be adopted; for background regions or low-priority global context regions, a relatively lower retention priority can be adopted. Furthermore, the retention priority is achieved by allocating different token budgets to each region unit: higher-priority region units are allocated more token budgets, retaining more visual representation units or generating more summary visual representation units; lower-priority region units are allocated less token budgets, retaining only a few summary visual representation units. Therefore, a first allocation rule can be set such that the token budget for target region units is higher than the token budget for global context regions, so as to achieve the goal of prioritizing the retention of target entity-related regions through the allocated token budget.

[0203] For the second allocation rule: using the target task requirement type as the analysis benchmark, a first term budget and a second term budget are allocated. For example, if the target task requirement type emphasizes local details (i.e., the target task requirement type is the first type), a larger proportion of the budget is allocated to the target entity-related region (i.e., the target region unit). If the target task requirement type emphasizes global semantics (i.e., the target task requirement type is the second type), a larger proportion of the budget is allocated to the global context region. For example, the first type of requirement type belongs to the type that focuses on local details (such as detailed regions, salient regions, and structurally dense regions), and the second type of requirement type belongs to the type that focuses on global semantics (such as main regions and background regions).

[0204] In this embodiment, a first lexical budget is set for the target region unit and a second lexical budget is set for the global context region. The first and second lexical budgets are obtained by allocating the total lexical budget according to a first allocation rule and / or a second allocation rule. Therefore, the visual representation units in the obtained target region unit and the visual representation units in the global context region can be compressed according to the first and second lexical budgets respectively to obtain initial visual representation information. This embodiment effectively controls the total number of lexical units in the initial visual representation information while ensuring the adaptability of the initial visual representation information to the image analysis task.

[0205] In one possible embodiment, predicting whether the inference result of the target time step belongs to a risky outcome based on the intermediate inference information of the target time step may include:

[0206] Step C1: Based on the intermediate inference information of the target time step, determine at least one evaluation value; wherein different evaluation values ​​are used to predict the unreliability of the inference result of the target time step from different prediction dimensions.

[0207] Step C2: Based on the at least one evaluation value, predict the answer risk index of the reasoning result for the target time step;

[0208] Step C3: If the answer risk index is higher than the predetermined risk threshold, the reasoning result of the target time step is determined to be a risky result.

[0209] For example, the intermediate inference information of the target time step may include the output probability of each candidate word, that is, the intermediate inference information of the target time step is the output probability distribution.

[0210] Furthermore, after obtaining the intermediate inference information for the target time step, at least one evaluation value can be determined to predict the unreliability of the inference result for the target time step from different prediction dimensions (also known as predicting instability). Then, based on the at least one evaluation value, the answer risk index of the inference result for the target time step is predicted through a predetermined calculation method (such as weighted fusion). It is understood that in this embodiment, the answer risk index is calculated based on the output state of the target time step. The answer risk index is used to characterize the instability, uncertainty, or unreliability of answer generation under the current visual evidence support.

[0211] When the answer risk indicator is higher than the predetermined risk threshold (via When representing the result, the reasoning result of the target time step is determined to be a risky result; that is, when the answer risk index is used... With the predetermined risk threshold Compare the following conditions:

[0212] ;

[0213] If the current visual context is deemed insufficient to support stable answer generation, a visual verification process is triggered. If the above conditions are not met, the current visual context is maintained, and the subsequent decoding process continues.

[0214] Any evaluation value that reflects the unreliability of the inference result at the target time step can be applied to the embodiments of this application, and this application does not limit it. For clarity and layout, the specific type of the at least one evaluation value will be described in the embodiments below. Furthermore, the predetermined risk threshold can be set according to actual circumstances and is not limited here.

[0215] In this embodiment, based on the intermediate inference information of the target time step, at least one evaluation value is determined to quantify the risk of the inference result. Based on this evaluation value, an answer risk index for the inference result of the target time step is predicted. Furthermore, if the answer risk index is higher than a predetermined risk threshold, the inference result of the target time step is determined to be a risky result. Therefore, this embodiment can effectively predict whether the inference result of the target time step is a risky result, thus providing a foundation for improving response accuracy.

[0216] In one possible embodiment, determining at least one evaluation value based on the intermediate inference information of the target time step includes:

[0217] Based on the intermediate inference information of the target time step and the target task requirement type of the image analysis task, at least one evaluation value is determined;

[0218] The at least one evaluation value may include:

[0219] At least one of the following: first evaluation value, second evaluation value, third evaluation value, fourth evaluation value, and fifth evaluation value;

[0220] Wherein, the first evaluation value represents the entropy of the probability distribution of the output probability of each candidate word;

[0221] The second evaluation value represents the difference between the highest output probability and the second highest output probability among the output probabilities of each candidate word.

[0222] The third evaluation value represents the number of output probabilities that are greater than a preset probability threshold among the output probabilities of each candidate word.

[0223] The fourth evaluation value represents the highest output probability among the output probabilities of each candidate word;

[0224] The fifth evaluation value represents the prior risk weight corresponding to the target task requirement type of the image analysis task.

[0225] Specifically:

[0226] The first evaluation value can be called: the entropy of the output probability distribution at the target time step, through... Characterization is performed, where t represents the target time step; for example, ,in, For the first The output probability of each candidate token (i.e., candidate word element) is such that a higher entropy value indicates that the model is more uncertain about the current output.

[0227] The second evaluation value is the difference between the highest and second-highest output probabilities among the output probabilities of each candidate word (also known as the probability interval). To characterize; for example, ,in and These are the output probabilities of the candidate tokens with the highest and second-highest probabilities, respectively. The smaller the interval, the less certain the model is about the candidate token ranked first.

[0228] The third evaluation value represents the number of output probabilities greater than a preset probability threshold among the output probabilities of each candidate word, that is, the number of candidate tokens exceeding the preset probability threshold in the output probability distribution of the target time step; the third evaluation value can be obtained through... To characterize; for example, ,in For the preset probability threshold, The more candidate tokens there are, the more dispersed the output distribution and the higher the uncertainty.

[0229] The fourth evaluation value represents the highest output probability among all candidate tokens, that is, the output probability value of the candidate token ranked first at the target time step. The lower the value, the lower the confidence level of the large language model in the current output and the higher the risk.

[0230] The fifth evaluation value represents: the prior risk weight corresponding to the target task requirement type of the image analysis task; for example, based on the target task requirement type (which can be determined by...). (Characteristics) allows querying a pre-defined risk prior weight table to obtain weight values. This yields the fifth evaluation value. For example, requirements for local detail recognition and counting correspond to higher prior risk weights, while requirements for scene semantics correspond to lower prior risk weights.

[0231] It should be noted that the first, second, third, and fourth evaluation values ​​are all calculated based on the intermediate inference information (i.e., the output probabilities of each candidate word) at the target time step. However, the fifth evaluation value is not based on the output probability of the current time step; it is determined by querying a preset risk prior weight table based on the target task requirement type parsed from the image analysis task before inference begins. Therefore, the fifth evaluation value can be reused at each target time step without step-by-step calculation. Thus, the at least one evaluation value may include only the evaluation value obtained based on the output probability (at least one of the first to fourth evaluation values), or it may combine the first evaluation value with the fifth evaluation value, or only the fifth evaluation value may be used. This application does not limit this.

[0232] Based on the above descriptions of the various evaluation values, when at least one evaluation value includes a first evaluation value, a second evaluation value, a third evaluation value, a fourth evaluation value, and a fifth evaluation value, the answer risk indicator... It can be represented as:

[0233] ;

[0234] in, The risk fusion function can be, for example, a weighted summation, a weighted product, or other combinations thereof. The specific content of the function is not specifically limited in this application.

[0235] In this embodiment, the at least one evaluation value includes at least one of a first evaluation value, a second evaluation value, a third evaluation value, a fourth evaluation value, and a fifth evaluation value, so that the output probability distribution can be effectively utilized to accurately predict the unreliability of the inference result of the target time step from at least one prediction dimension; thereby, providing a basis for further improving the response accuracy of image analysis tasks.

[0236] In one possible embodiment, if the visual representation set of the target image is cached after its construction, the supplementary processing of the visual context may include:

[0237] Step D1: Based on the retrieval information, select a portion of the visual representation units from the cached visual representation set;

[0238] Step D2: Based on the selected visual representation units, generate target supplementary information for the image analysis task;

[0239] Step D3: Add the target supplementary information into the current visual context;

[0240] The retrieval information includes:

[0241] The image analysis task, the target feature information, the currently generated inference result sequence of the large language model, the target task requirement type of the image analysis task, and the answer risk indicator are at least one of the following: the answer risk indicator is used to analyze whether a result is risky.

[0242] In this embodiment, after the visual supplementation process is triggered (that is, after the process of supplementing the visual context is triggered), based on the retrieval information, a portion of the visual representation units can be selected from the cached visual representation set, that is, the visual evidence to be supplemented can be located; based on the selected visual representation units, the target supplementary information of the image analysis task is generated; and the target supplementary information is added to the current visual context.

[0243] Among them, the selected visual representation units can be directly used as target supplementary information for the image analysis task; alternatively, the selected visual representation units can be compressed, and the information obtained from the compression process can be used as target supplementary information.

[0244] It should be noted that, based on the retrieved information, there can be multiple ways to select a portion of the visual representation units from the visual representation set.

[0245] Optionally, in one implementation, the step of filtering a portion of the visual representation units from the cached visual representation set based on retrieval information may include:

[0246] Step D11: Based on the retrieval information, select a region unit from the region set containing multiple region units corresponding to the cached visual representation set to obtain a supplementary unit; wherein, each region unit contains a visual representation unit in the visual representation set whose corresponding sub-region belongs to the same local region of the image.

[0247] Step D12: Obtain the visual representation units in the supplementary unit from the cached visual representation set, so as to filter and obtain a portion of the visual representation units.

[0248] In this implementation, firstly, based on the retrieved information, the region units are selected, that is, the region units are located to obtain the supplementary evidence units; then, from the cached visual representation set, the visual representation units in the supplementary evidence units are obtained to filter and obtain a portion of the visual representation units; thus, the filtered portion of the visual representation units can be obtained quickly.

[0249] Furthermore, based on the retrieved information, the selection of region units from the region set containing multiple region units corresponding to the cached visual representation set can be achieved in various ways to obtain the supplementary evidence unit.

[0250] For example, when the target feature information includes the target entity category, a combination of the first and second strategies can be used to determine the supplementary verification unit. The first strategy prioritizes filtering supplementary verification units in areas related to the target entity. The second strategy selects the corresponding type of area unit based on the matching relationship between the task requirement type and the area type metadata (i.e., the correspondence between each task requirement type and its metadata described below). Accordingly,

[0251] The step of selecting a region unit from a region set containing multiple region units corresponding to the cached visual representation set based on retrieval information to obtain a supplementary verification unit may include:

[0252] Step D111: Based on the target entity category, select region units from the region set containing multiple region units corresponding to the cached visual representation set to obtain the first region set;

[0253] Step D112: Based on the target task requirement type of the image analysis task and the correspondence between each task requirement type and metadata, select region units from the obtained first region set to obtain a second region set; wherein, the metadata corresponding to each task requirement type is: the metadata required for the region unit to be selected under that task requirement type.

[0254] Step D113: Select a supplementary proof unit from the second region set.

[0255] Wherein, when the target feature information includes the target entity category, a first region set can be obtained by selecting region units whose entity category labels match the target entity category and meet predetermined conditions from the region set corresponding to the visual representation set, which contains multiple region units; the predetermined conditions include: the included visual representation units have not been used to generate initial visual representation information or target supplementary information that has been generated.

[0256] Furthermore, using the target task requirement type of the image analysis task as a further screening criterion, and combining the correspondence between each task requirement type and metadata, region units are selected from the obtained first region set to obtain a second region set; wherein, the metadata corresponding to each task requirement type is: the metadata required for the region unit to be selected under that task requirement type.

[0257] After obtaining the second region set, all or part of the region units can be selected from the second region set to obtain the supplementary verification units. In other implementations, a subset can be selected from the second region set according to a predetermined subset selection method to obtain a unit set, and a preset number of region units or those meeting budget constraints can be selected from this unit set to obtain the supplementary verification units. The predetermined subset selection method can be set according to actual conditions, such as a selection method based on the degree of matching with the image analysis task, a selection method based on whether the region unit itself has a specific metadata, etc. This application does not specifically limit this method.

[0258] It should be emphasized that the above steps D111-D113 are merely examples and should not be construed as limiting the embodiments of this application. Exemplarily, the criteria for locating the target supplementary information (i.e., the selection strategy for the supplementary evidence unit) include, but are not limited to:

[0259] Prioritize filtering region units (i.e., correction units) in the target entity-related regions for use as supplementary evidence; establish a matching relationship between task requirement type and region type metadata, and select region units with corresponding region type metadata according to different task requirement types; select region units whose feature information matches the keywords of the image analysis task (e.g., a question); select object regions, text regions, or attribute regions that are relevant to the currently generated content (i.e., region units whose feature information satisfies the similarity condition between the vector of the currently generated content); pre-defined high-risk detail regions (e.g., region units whose internal visual feature changes exceed a set threshold, etc.); regions that have not yet been injected into the current visual context but have high potential information value (e.g., region units whose feature differences with neighboring region units are greater than a set threshold, etc.); and region units corresponding to the candidate token with the lowest confidence in the current output.

[0260] To better understand the process of supplementing visual context processing, the following will describe it from the perspective of a complete process:

[0261] In this implementation process, based on the image analysis task (such as a question), the currently generated output sequence (i.e., the inference result sequence currently generated by the large language model), the target entity category, the task requirement type, the answer risk index (also known as the answer risk status), and visual index information (information used to index region units), the most relevant supplementary evidence unit for the current answer generation is located from the visual cache using a target entity-first strategy; the set of supplementary evidence units formed by these supplementary evidence units can be represented as:

[0262] ;

[0263] in, This represents the function for locating the supplementary evidence region, i.e., the function used to determine the set of supplementary evidence units; Represents visual index information, The output sequence that has been generated so far is T, where T is the image analysis task. This is a risk indicator for the answer.

[0264] Furthermore, extract the set of supplementary evidence units from the visual cache. The corresponding supplementary set of visual evidence (i.e., the set formed by the visual representation units selected in step D1):

[0265] ;

[0266] in, This represents the set of supplementary visual evidence to be injected into the current context. These are the visual representation units cached in the visual cache.

[0267] Specifically: After detecting that the answer risk indicator meets the trigger condition (exceeding the predetermined risk threshold), the process of locating the target supplementary information (i.e., the process of supplementing the visual context) is executed to determine the supplementary verification unit related to the current answer generation from the visual cache, and to extract the visual representation unit within the supplementary verification unit for updating the current visual context. For example... Figure 3 As shown, the target supplementary information location process may include:

[0268] S301, the current target for supplementary certification has been identified.

[0269] S301 is used to determine the supplementary evidence strategy / supplementary evidence requirement, for example: based on text input. (i.e., image analysis task), currently generated output sequence Target entity category and task requirement type Determine the current goals for obtaining supplementary certification.

[0270] Optionally, when the text input T involves a specific target entity category, supplementary evidence is first performed around the relevant area corresponding to the target entity category; when the text input T does not involve a specific target entity category, supplementary evidence is first performed around the main area, the salient area, or the global context area.

[0271] S302, Candidate region selection with priority given to target entities.

[0272] First, based on the target entity category Based on the entity relevance information (such as entity category labels) of each regional unit, a set of target entity-related regions (i.e., the first region set mentioned above) is selected from multiple regional units:

[0273] ;

[0274] in For the set of regions related to the target entity, This represents the matching function for entity relevance information of a region unit and the target entity category.

[0275] For example, when the target entity is classified as a person, vehicle, or pet, subsequent supplementary evidence candidate screening is performed preferentially in the corresponding person-related area, vehicle-related area, or pet-related area.

[0276] For example, when the set of regions related to a target entity is empty, insufficient in number, or the category of the target entity cannot be clearly defined, candidate regions can be supplemented from the main region, salient region, or global context region.

[0277] Subsequently, based on the type of target task requirements With the region type metadata of each region unit (i.e. The matching relationship between the target entity and the relevant region set is used to filter the candidate unit set (i.e., the second region set mentioned above):

[0278] ;

[0279] in For the candidate unit set, This function represents the matching function between the target task requirement type and the region type metadata.

[0280] For example, when filtering the candidate unit set from the set of regions related to the target entity, a predetermined region unit retention strategy in the above embodiments can be referred to. This predetermined region unit retention strategy includes: a mapping relationship between each task requirement type and region retention rules; wherein, the mapping relationship between each task requirement type and region retention rules can be as follows:

[0281] For attribute-based requirements or local detail recognition requirements, the mapping is: prioritize preserving areas with rich details and main body areas;

[0282] The requirements mapping for counting types includes: prioritizing the preservation of densely structured regions;

[0283] Relationship-based requirement mappings include: prioritizing the retention of the main area and the spatial relationship-sensitive areas of multiple entities;

[0284] Subject recognition requirements include: prioritizing the preservation of the main subject area and areas rich in detail;

[0285] The scene semantic class requirement mapping includes: prioritizing the retention of the main area and a predetermined number of global context areas.

[0286] S303, Candidate region sorting and supplementary evidence region determination.

[0287] For candidate unit set The relevance of each regional unit (which can be called a candidate region) is ranked to select the set of supplementary evidence units. .

[0288] For example, for the candidate unit set When ranking the relevance of each regional unit in the data, at least one of the following information can be used:

[0289] The degree of matching between the regional feature representation (i.e., the characteristic information of regional units) and the semantic representation of the text input;

[0290] The degree of matching between the target task requirement type and the region type metadata;

[0291] Regional salience or priority information;

[0292] The correlation strength between regional units and target entity categories.

[0293] Furthermore, a preset number of regional units or units that meet budget constraints are selected from the candidate unit set to obtain a supplementary verification unit set (i.e., a set of supplementary verification units). ;in, , This represents the candidate region selection function (i.e., the function used to select the supplementary proof unit).

[0294] S304, extract target supplementary information from visual cache.

[0295] According to the set of supplementary evidence units Furthermore, regarding the mapping relationship between region units and visual representation units (such as visual index information), the corresponding target supplementary information (i.e., each visual representation unit within the supplementary unit) is extracted from the visual cache.

[0296] ;

[0297] For example, after obtaining the target supplementary information, the target supplementary information can be filtered according to the current budget constraints to obtain the final target supplementary information, which is then used as the target supplementary information. Subsequently, the target supplementary information is incorporated into the current visual context for use in the subsequent decoding process of the multimodal large language model.

[0298] In this embodiment, based on the retrieved information, a subset of visual representation units are selected from the visual representation set; based on the selected visual representation units, target supplementary information for the image analysis task is generated; and the target supplementary information is added to the current visual context. Through this embodiment, effective target supplementary information can be selected, enabling effective updating of the visual context and increasing the information content of the visual context.

[0299] In one possible implementation, the content in the current visual context is cached in key-value format;

[0300] The step of adding the target supplementary information into the current visual context includes:

[0301] The information content of the target supplementary information is added to the current visual context in the form of key-value pairs;

[0302] or,

[0303] From the current visual context, identify content that meets the predetermined removal conditions, and replace the identified content with the information content of the target supplementary information in the form of key values.

[0304] In this embodiment, adding the target supplementary information into the current visual context can be characterized as follows:

[0305] ;

[0306] in, For the updated visual context, For the current visual context, Supplement the target with information (i.e., supplement visual evidence).

[0307] Understandably, after obtaining the updated visual context, the updated visual context can be... Together with image analysis tasks (such as text output representing a problem) and historical output sequences (output sequences or partial answer content generated over historical time steps), they form an enhanced multimodal context for subsequent answer generation.

[0308] Optionally, in one implementation, the target supplementary information is added to the current visual context in key-value pairs. That is, the injection of target supplementary information adopts an append-only mode. For example, the target supplementary information is appended to the end of the current key-value cache (KV cache) of the large language model. Specifically, only one incremental forward computation is needed to generate the corresponding key-value pair for the target supplementary information, and the key-value pair is appended to the existing key-value cache, without recalculating the key-value cache of the existing sequence. Therefore, in subsequent decoding time steps, the attention mechanism of the large language model can simultaneously focus on the initial visual representation information and the target supplementary information. It should be noted that the single supplementary computation cost of the append-only mode is... ,in The number of visual representation units in the supplementary information for the target. The hidden layer dimension of the model; after the supplementary proof, the attention computation length of each subsequent decoding time step increases. .

[0309] Optionally, in another implementation, content that meets predetermined rejection conditions is identified from the current visual context, and the information content of the target supplementary information is used as a key value to replace the identified content. That is, the injection of the target supplementary information adopts a replacement mode. For example, the visual representation unit with the lowest information value (obtained through a set evaluation method) is identified from the current visual context, and its corresponding key value is replaced with the key value of the target supplementary information. It should be noted that the replacement mode does not increase the total sequence length, and therefore does not increase the attention computation overhead of subsequent decoding time steps.

[0310] It should be noted that the visual verification process is triggered only when the answer risk index exceeds a threshold, and not executed at every decoding time step. In most decoding time steps, only the answer risk index needs to be calculated (obtained directly from the existing output probability distribution, with negligible computational overhead), and the verification process is not triggered, thus incurring no additional visual context update overhead.

[0311] In this embodiment, when the target supplementary information is added to the current visual context, an append mode or a replacement mode can be used to ensure the efficiency and effectiveness of updating the visual context.

[0312] In one possible embodiment, the supplementary processing of the visual context may include:

[0313] Before the inference results for this target time step are generated, supplementary processing of the visual context is performed;

[0314] Accordingly, the method further includes:

[0315] Trigger the backoff operation of the large language model for the target time step;

[0316] or,

[0317] The supplementary processing of the visual context includes:

[0318] Before the specified time step for startup is reached, perform supplementary processing of the visual context;

[0319] The specified time step is either the next time step after the target time step, or the time step used to generate the specified answer segment for the image analysis task.

[0320] In this embodiment, target supplementary information (i.e., supplementary visual evidence) can be incorporated into the multimodal context at any of the following times: before the current decoding time step is recalculated; before the next decoding time step begins; and before the predicted specified answer segment is generated.

[0321] It is understandable that, depending on whether the selected output token at the current time step (i.e., the current target time step) is retained, there are two types of injection opportunities for the target supplementary information:

[0322] For the first type of injection timing, the target supplementary information is applied to the output token at the current time step. If the answer risk exceeds a threshold after the output probability distribution at the current time step has been calculated but before the output token is selected, the target supplementary information is first incorporated, based on the updated visual context. The output probability distribution at the current time step is recalculated and the output token is selected accordingly (i.e., the backtracking operation of the large language model for the target time step is triggered), thereby avoiding the selection of unreliable output tokens when visual evidence is insufficient.

[0323] For the second type of injection timing, the selected output token at the current time step is retained, and the target supplementary information is applied to the output tokens at subsequent time steps. This type of injection timing does not require recalculating the current time step, resulting in lower decoding latency. Specifically, it can include:

[0324] (i) Incorporate before the start of the next decoding time step so that subsequent outputs can focus on the target supplementary information (corresponding to the specified time step above: the next time step of the target time step).

[0325] (ii) When it is predicted that a specified answer segment will be generated subsequently, it is incorporated before the output token corresponding to the specified answer segment is selected, in order to ensure the generation quality of the specified answer segment (corresponding to the specified time step mentioned above: the time step used to generate the specified answer segment of the image analysis task). For example, the specified answer segment can be a pre-defined answer fragment that requires key quality assurance (which can be called a key answer segment), such as an output fragment carrying the core content of the answer, such as attribute words, counting results, entity names, or recognition results; its prediction method may include: according to the image analysis task and the target task requirements type. The answer structure can be analyzed to determine its expected location, or the location can be determined based on the upward trend of the answer risk indicator, or the location can be predicted based on the output sequence generated from the initial compact visual representation.

[0326] In this embodiment, visual context supplementation is performed according to the set injection timing, thereby ensuring that the updated visual context can be effectively utilized.

[0327] To better understand the method provided in the embodiments of this application, the execution subject of the method provided in the embodiments of this application is described below from the perspective of functional modules: the response device for image analysis tasks.

[0328] like Figure 4 As shown, the response device for an image analysis task may include:

[0329] Input acquisition module 410: used to acquire the target image (i.e. Figure 4 Images in the image) and image analysis tasks (i.e. Figure 4 The text T in the middle.

[0330] Visual encoding and caching module 420: used to perform visual encoding on the target image, generate a complete set of visual representations (the aforementioned set of visual representations), and establish visual caching and visual index information.

[0331] The initial visual compression module 430 is used to generate initial visual representation information from the complete visual representation set and input the initial visual representation information into the large language model. The initial visual compression module may include a text parsing submodule, a region organization submodule, an entity-related region identification submodule, and a target entity priority compression submodule. Through the interaction of these submodules, the generation of initial visual representation information can be achieved.

[0332] Multimodal decoding module 440: namely the large language model, used to perform answer generation based on the image analysis task and the current visual context, and to generate the final output answer for the image analysis task.

[0333] Risk assessment and triggering module 450: used to calculate the answer risk index based on the current decoding status, and determine whether to trigger supplementary verification based on the comparison result of the answer risk index and the threshold.

[0334] The supplementary evidence localization and injection module 460 is used to locate supplementary target information based on the image analysis task, generated answer content, risk status, and visual index information, and to extract supplementary target information from the visual cache and update the visual context. The supplementary evidence localization module may include a supplementary evidence target determination submodule, a target entity priority candidate region filtering submodule, a candidate region sorting submodule, and a cache extraction submodule. Through the interaction of these submodules, supplementary target information can be obtained.

[0335] Adaptive inference control module (not shown in the figure): used to control the number of supplementary evidence attempts, supplementary evidence budget, decoding pace and stopping conditions.

[0336] Through the interaction of the various modules described above, the image analysis task response method provided in this application embodiment can be realized, thereby balancing the response efficiency and accuracy for image analysis tasks.

[0337] To better understand the method provided in this application, the following will be combined with... Figure 5 The overall flow of a response method for an image analysis task provided in the embodiments of this application will be introduced.

[0338] like Figure 5 As shown, a response method for an image analysis task may include:

[0339] S501: Acquire input data; that is, acquire image data and text input.

[0340] S502: Constructing a complete visual representation and visual cache; that is, the process of performing visual encoding, generating a complete visual representation (i.e., the set of visual representations mentioned above), and visual cache;

[0341] S503: Generate initial visual representation information; that is, perform text parsing, region organization, and target entity priority compression processes.

[0342] S504: Perform initial multimodal inference; that is, perform the process of decoding based on text input and initial visual representation information through a large language model;

[0343] S505: Calculate the answer risk index; that is, calculate multiple evaluation values: output entropy, probability interval, candidate dispersion, confidence, task risk weight, and calculate the answer risk index based on the calculated evaluation values;

[0344] S506: Does the answer risk indicator exceed the threshold? If yes, trigger S507; if no, trigger S509.

[0345] S507: Locating supplementary information for the target; that is, the process of performing priority candidate region filtering and sorting of the target entity and extracting visual representation units from the visual cache;

[0346] S508: Update visual context; incorporate target supplementary information into the current visual context;

[0347] S509: Continue with answer generation;

[0348] Furthermore, it can detect whether the stopping condition is met, i.e., execute S510. If yes, output the final answer; otherwise, continue to trigger S505.

[0349] Optionally, the initial visual representation information (i.e., the initial compact visual representation set) is not directly selected from the complete visual representation units, but is obtained through region-level summarization, hierarchical compression, or multi-scale aggregation to avoid inputting too many fine-grained visual representation units into the large language model in the initial stage.

[0350] Optionally, the answer risk index can be calculated not only based on the probability distribution of a single-step output, but also by combining the risk prior weights corresponding to the task requirement type and the candidate dispersion of the current time step.

[0351] Optionally, the injection method for the target supplementary information includes:

[0352] Append mode: The key-value pairs corresponding to the target supplementary information are appended to the end of the current key-value cache of the large language model. Only incremental forward calculation is required for the target supplementary information, without affecting the existing key-value cache.

[0353] Replacement mode: Replace the key-value pair of the visual representation unit with the lowest information value in the current visual context with the key-value pair of the target supplementary information, without increasing the total length of the sequence;

[0354] Additionally, supplementary target information is incorporated into the multimodal context (i.e., visual context) in the form of a region summary.

[0355] Use supplementary information about the target as dedicated visual evidence input for the current or subsequent decoding steps.

[0356] Optionally, to control reasoning overhead, the maximum number of rounds of supplementary evidence can be limited. Upper limit of visual representation unit for single supplementary certification and total visual budget And if the risk-reward ratio is insufficient, further supplementary certification will cease. When using the supplementary certification model, the upper bound of the total computational overhead introduced by supplementary certification is... The incremental forward computation overhead, plus the increased attention computation length in each subsequent decoding time step, adds up to a maximum of One position. When Much smaller than the number of visual representation units initially compressed. At that time, the total cost of supplementary verification is much less than the cost of inputting all visual representation units at the beginning.

[0357] As can be seen, the image analysis task response method provided in this application can reduce the input size of visual representation units and the inference overhead during the multimodal large language model inference process, while ensuring the accuracy and stability of answer generation. Its core idea is as follows: For the input image, instead of injecting all visual representation units into the large language model at the initial stage, a complete visual representation is first constructed and cached. Based on the target entity category and task requirement type parsed from the text input, visual representations of target entity-related regions are prioritized and retained. This generates an initial compact visual representation set much smaller than the complete visual representation, which is then input into the large language model, thereby reducing the computational load and key-value cache usage during initial inference. During answer generation, the answer risk in the current decoding state is evaluated in real time. When uncertainty or insufficient stability is detected in the current answer, supplementary target information related to the current question and generation state is located and extracted from the pre-established visual cache using a target entity-first strategy. This supplementary target information is then incorporated into the current multimodal context (i.e., visual context) to support subsequent answer generation. Thus, inference is completed with fewer visual representation units in most decoding time steps, with supplementation only as needed, achieving a balance between inference acceleration and answer quality.

[0358] To better understand how the initial visual representation information is generated (i.e., how the initial compact visual representation set is generated), the following will combine... Figure 6 Let me introduce it.

[0359] like Figure 6 As shown, the process of generating initial visual representation information includes:

[0360] S601: Text parsing; that is, after obtaining text input, parsing the target entity category and task requirement type;

[0361] S602: Determine the regional organization and entity-related regional identifiers; that is, perform the process of dividing regions, generating regional features and metadata, and identifying the target entity-related regions.

[0362] S603: Generation of initial visual representation information prioritizing target entities; that is, determining retention priorities based on target entity categories, task requirement types, regional metadata, and budget, and generating initial visual representation information accordingly.

[0363] S604: Visual cache establishment; that is, caching the complete visual representation and mapping based on region (i.e., region unit) - cache (i.e., cache location).

[0364] After obtaining the initial visual representation information, the initial visual representation information can be fed into the large language model; and the visual cache is used for candidate supplementary evidence.

[0365] To understand the image analysis task response method provided in the embodiments of this application, a specific embodiment will be described below.

[0366] The response method for this image analysis task is applied to a multimodal reasoning system (i.e., an implementation of the response device for the aforementioned image analysis task) that includes a visual encoder, a visual cache module, a text parsing module, and a large language model decoding module. Furthermore, this method accelerates multimodal reasoning by prioritizing the supplementary evidence of target entities. It is also applicable to task scenarios such as image question answering, visual attribute recognition, target counting, and fine-grained recognition.

[0367] In this embodiment, the input is an image to be processed. (Corresponding to the target image mentioned above) and the corresponding question text. (Corresponding to the image analysis task described above); accordingly, the response method for the image analysis task may include the following steps.

[0368] Step 1: Generate a complete set of visual representations.

[0369] In this step, the image Input the visual encoder to obtain the complete set of visual representations (corresponding to the visual representation set mentioned above):

[0370] ;

[0371] in, Indicates the first Each visual representation unit (also understood as the visual feature corresponding to the i-th visual representation unit). This represents the total number of visual representation units. In this embodiment, the visual encoder can be a patch-based visual coding network, where the image is divided into multiple local regions and encoded as visual representation units; simultaneously, the relationship between each visual representation unit and the image is recorded. The mapping relationship between local regions includes the spatial location index of the visual representation unit and the information of the region to which it belongs.

[0372] Step 2: Obtain the target entity category and target task requirement type through text parsing.

[0373] In this step, the question text will be... Input text parsing module to obtain the target entity category corresponding to the current question. and target task requirements type This process can be represented as:

[0374] ;

[0375] In multi-turn question-and-answer scenarios, if the current question-and-answer process pertains to an image... So, the first round of Q&A process, It can be null; otherwise, For images Reference content (such as questions, output answers, etc.) under the historical question-and-answer rounds.

[0376] For example, the text parsing module can use keyword rule parsing to extract the target entity category and the target task requirement type. For instance, when the question text... When the target entity contains keywords such as "person", "man", "woman", and "people", the target entity category will be determined. The parsing is "person"; when the question text When the target entity contains keywords such as "car", "bus", "truck", and "vehicle", the target entity category will be determined. Parsed as "vehicle"; when the question text When the target entity category contains keywords such as "dog", "cat", and "pet", the target entity category will be determined. The interpretation is "pet".

[0377] And, when the question text When keywords such as "color", "wearing", and "material" are included, the target task requirement type should be considered. Parsing into attribute class requirements; when the question text When the target task requires keywords such as "how many" or "number of", the task type should be specified. Parsing it as a counting requirement; when the question text When keywords such as "what type", "what kind", and "which breed" are included, the target task requirement type should be specified. The analysis is for subject recognition requirements; when the question text... When the target task requires keywords such as "left", "right", "next to", or "behind", the task type should be specified. Parsing it as a relational requirement; when the question text When keywords such as "what is written", "number plate", and "license plate" are included, the target task requirement type should be specified. This is interpreted as a requirement for identifying local details.

[0378] Understandably, when it is impossible to obtain information from the question text... When explicitly resolving the target entity category, the target entity category can be set to an unknown category.

[0379] Step 3: Region division and region feature generation.

[0380] In this step, based on the visual representation unit and the image The mapping relationship between local regions will represent the complete visual representation set. It is divided into multiple regional units.

[0381] Specifically, the image The corresponding visual representation units are divided into two-dimensional grids. From the given regions, we obtain a set of regions:

[0382] ;

[0383] For each region Determine the corresponding subset of visual representation units:

[0384] ;

[0385] And the region feature representation (corresponding to the feature information of the region unit mentioned above) is obtained through aggregation operation:

[0386] ;

[0387] The aggregation operation is implemented using the mean pooling method.

[0388] Step 4: Generate region type metadata and entity-related information.

[0389] For details on the implementation process, please refer to the relevant content of the foregoing embodiments, which will not be repeated here.

[0390] Step 5: Generate initial visual representation information; that is, generate an initial compact visual representation set.

[0391] In this step, based on the target entity category Target task requirements type Region type metadata and budget constraints Generate initial visual representation information The initial visual representation information includes two parts:

[0392] ;

[0393] in, This is the visual representation obtained after summarizing the relevant regions of the target entity (corresponding to the information after compressing the visual representation units in the target region units mentioned above). This refers to the visual representation obtained after compressing the context region (corresponding to the information after compressing the visual representation units in the global context region as described above).

[0394] For target entity-related regions (regional units whose entity category labels match the target entity category), a higher retention priority is applied; for background regions, only a small amount of contextual summary is retained.

[0395] And, setting budget constraints The upper limit of the total number of visual representation units for the initial visual representation information; budget constraints. Proportional allocation to the target entity-related region and the global context region: Target entity-related region allocation Token budget, global context area allocation A token budget, of which This is the preset budget ratio for the physical area.

[0396] For each region unit in the target entity's relevant region, the number of visual representation units to be retained is determined based on its region type metadata and the target task requirement type. Specifically, for high-priority region units, a subset of representative visual representation units are selected from the visual representation unit subset of that region unit and directly retained; for low-priority region units, the subset of visual representation units of that region unit is averaged and pooled to generate a region-level summary visual representation unit.

[0397] The region preservation strategies for each task requirement type are as follows: When the task requirement type is attribute-based or local detail recognition-based, priority is given to preserving detailed regions and main body regions related to the target entity, retaining more original visual representation units for these regions; When the task requirement type is counting-based, priority is given to preserving structurally dense regions, ensuring that each instance's region is covered by visual representation units; When the task requirement type is relationship-based, priority is given to preserving the main body region and spatial relationship-sensitive regions containing multiple entities; When the task requirement type is subject recognition-based, priority is given to preserving the main body region and detailed regions; When the task requirement type is scene semantic-based, priority is given to preserving the main body region and a small amount of context regions, while only the average pooled summary visual representation units are retained for the background region.

[0398] This forms the initial visual representation information. And input it into a large language model for initial decoding.

[0399] Step 6: Establish visual cache and the mapping relationship between region units and cache.

[0400] In this step, the complete visual representation set is... Stored in visual cache:

[0401] ;

[0402] Simultaneously, record the set of indices of the cached visual representation units corresponding to each region unit:

[0403] ;

[0404] The visual cache, and the mapping relationship between region units and the cache (i.e., the index set of cached visual representation units corresponding to each region unit) are used to extract supplementary visual representation units when subsequent supplementary certification is triggered.

[0405] Step 7: Perform initial decoding and calculate the answer risk index.

[0406] Question text and initial visual representation information Input a large language model and perform answer generation. At each decoding time step (i.e., each time step mentioned above), calculate the answer risk index based on the current output probability distribution. The answer risk index can be determined by at least one of the following evaluation values:

[0407] The system determines the answer risk index by considering at least one of the following: output entropy, the probability interval between the first and second ranked candidate tokens, the number of candidate tokens exceeding a preset probability threshold, and the output probability value of the first ranked candidate token. When the answer risk index exceeds the preset threshold, a visual verification process is triggered.

[0408] Step 8: Filter candidate regions with priority given to target entities.

[0409] After triggering visual verification, first determine the target entity category. Set of regions related to the target entity First, determine the scope of priority candidate regions for entities. Then, based on the target task requirement type... With region type metadata The matching relationship between them is derived from the set of relevant regions of the target entity. Select candidate unit set:

[0410] ;

[0411] In this embodiment, when For attribute-based or local detail recognition requirements, prioritize preserving areas rich in detail and the main body area; when When dealing with counting requirements, prioritize preserving densely structured areas; when When dealing with relationship-related needs, priority should be given to preserving the main area and areas sensitive to spatial relationships; when When the requirement is for subject recognition, priority should be given to preserving the subject area and areas with rich details; When dealing with scene semantic requirements, prioritize preserving the main body area and a small amount of global context area.

[0412] when If empty, supplement candidate regions from the salient region and the main region.

[0413] Step 9: Sort candidate regions and determine supplementary proof regions.

[0414] For candidate unit set The relevance of each regional unit in the model is ranked. Based on information such as the degree of matching between the regional feature representation and the semantic representation of the current problem, and regional saliency, regional units that meet the budget constraints are selected from the candidate unit set to obtain the supplementary proof unit set. (The set formed by the above-mentioned supplementary proof units).

[0415] Step 10: Extract supplementary target information from the visual cache and continue decoding.

[0416] According to the set of supplementary evidence units The mapping relationship between region units and cache is established, and the corresponding supplementary visual representation unit set (i.e., target supplementary information) is extracted from the visual cache.

[0417] ;

[0418] Understandably, if the current budget for supplementary certification is limited, further funding can be allocated from... The final set of supplementary visual representations that meets the budget constraints is selected. Then, the set of supplementary visual representations is incorporated into the current visual context, and the subsequent decoding of the large language model continues until the final answer is generated.

[0419] Through this embodiment, for the question text T used in the image analysis task, both response efficiency and response accuracy can be taken into account.

[0420] In addition, it should be noted that the image analysis task response method provided in this application has the following innovative points and technical effects:

[0421] Innovation Point 1: A dynamic visual verification mechanism based on answer risk during the reasoning process.

[0422] Unlike related technologies that use a static compression strategy to determine the visual representation unit and retain the result all at once before reasoning begins, this application calculates the answer risk index in real time during the answer generation process. When the risk exceeds the threshold, the target supplementary information is extracted from the visual cache as needed.

[0423] Beneficial effects: It tightly couples the scheduling of visual information with the answer generation process, supplementing visual details only when the large language model truly needs them, thus avoiding the problem of irrecoverable key visual information caused by static compression; it can complete reasoning without supplementary proof for simple problems, and can supplementary proof in multiple rounds for complex problems, achieving adaptive allocation of computing resources.

[0424] Innovation Point Two: Complete Visual Representation Caching and On-Demand Retrieval Architecture

[0425] Unlike related technologies that directly discard unselected visual representation units after compression, this application stores the complete set of visual representations in a visual cache and establishes a mapping relationship between region units and the cache, so that visual representation units of any region can be extracted from the cache at any time during the inference process.

[0426] Beneficial effects: Visual encoding only needs to be performed once, and subsequent supplementary verification processes do not require re-encoding, reducing computational overhead; visual information excluded by initial compression is not lost and can be recovered at any time when needed, improving the system's fault tolerance; when supplementary verification adopts append mode, only incremental forward computation needs to be performed on the supplementary visual representation units, without recalculating the key-value cache of the existing sequence, and the total supplementary verification overhead is controllable and much less than the overhead of inputting all visual representation units initially.

[0427] Innovation Point Three: Target Entity-Prioritized Visual Representation Compression and Supplementation Strategy

[0428] Unlike related technologies that use a uniform compression strategy for all visual regions, this application obtains the target entity category and target task requirement type through text parsing. In both the initial compression and supplementary localization stages, it prioritizes processing the target entity-related regions and performs differentiated scheduling based on the target task requirement type and matching region type metadata.

[0429] Beneficial effects: Prioritizes the retention of visual information of target entities most relevant to the problem in the initial stage, improving the quality of the initial reasoning answer; accurately locates key areas related to target entities in the supplementary evidence stage, avoiding unnecessary full visual supplementation and improving the efficiency of supplementary evidence; matches different types of regions (detailed regions, structurally dense regions, etc.) according to different task requirements (attribute-based, counting-based, relation-based, etc.), realizing visual scheduling for task perception.

[0430] Innovation Point Four: Multi-dimensional Answer Risk Assessment Mechanism

[0431] The answer risk index proposed in this application comprehensively considers multiple dimensions, including output probability distribution entropy, probability interval between candidate rankings, candidate dispersion, output confidence, and prior weight of task requirement type risk. The first four dimensions can be directly calculated from the output probability distribution within a single decoding time step, and the prior weight of task requirement type risk is determined by text parsing before inference begins. All dimensions do not require additional multiple sampling or repeated inference, and can comprehensively evaluate the reliability of answer generation supported by current visual evidence without increasing inference overhead.

[0432] Beneficial effects include the fact that all risk sub-indicators (i.e., the aforementioned assessment values) can be directly calculated from the current output probability distribution within a single decoding time step without introducing additional sampling or repeated inference overhead, which aligns with the goal of accelerating inference. Compared to a single indicator (such as using only output entropy), multi-dimensional risk assessment can more accurately identify key time steps that require visual verification, reducing false triggers and missed triggers.

[0433] Corresponding to the above method embodiments, this application also provides a response device for an image analysis task, such as... Figure 7 As shown, it may include:

[0434] The module 710 is used to obtain target feature information for the image analysis task.

[0435] The determination module 720 is used to determine the set of visual representations of the target image indicated by the image analysis task;

[0436] The generation module 730 is used to generate initial visual representation information for the image analysis task based on the target feature information and the visual representation set; wherein, the initial visual representation information is used to characterize: the visual information of a portion of the visual representation units in the visual representation set;

[0437] The module 740 is used to invoke a large language model to process the image analysis task, using the initial visual representation information as the initial visual context.

[0438] The analysis module 750 is used to predict whether the inference result of the target time step is a risky result based on the intermediate inference information of the target time step after the large language model generates the intermediate inference information of the target time step. If it is, it performs supplementary processing of the visual context.

[0439] Optionally, the generation module includes:

[0440] The determination submodule is used to determine the region set corresponding to the visual representation set, which contains multiple region units; wherein, each region unit contains a visual representation unit in the visual representation set whose corresponding sub-region belongs to the same local region of the image;

[0441] The selection submodule is used to select region units from the region set based on the target feature information to obtain target region units;

[0442] A construction submodule is used to construct the initial visual representation information for the image analysis task based on the visual representation units in the obtained target region units.

[0443] Optionally, the target feature information includes: the target entity category indicated by the image analysis task;

[0444] The selection submodule is specifically used for:

[0445] Determine the entity category label for each region unit; wherein, the entity category label for each region unit is used to characterize: the entity category of the local image region to which the visual representation unit within that region unit belongs;

[0446] Based on the entity category label of each region unit, a target region unit whose entity category label matches the target entity category is selected from the region set.

[0447] Optionally, each region unit is provided with metadata, and the metadata represents the region type of the local image region to which the visual representation unit within that region unit belongs;

[0448] The device further includes:

[0449] The context determination module is used to select a region unit with specified metadata from the region set based on the metadata of each region unit before constructing the initial visual representation information of the image analysis task based on the visual representation units in the obtained target region units, thereby obtaining a global context region; wherein, the region type represented by the specified metadata is a region type that can represent the scene semantics of the target image.

[0450] The construction submodule includes:

[0451] An information construction unit is used to construct initial visual representation information for the image analysis task based on the visual representation units in the obtained target region unit and the visual representation units in the global context region.

[0452] Optionally, the information construction unit is specifically used for:

[0453] The visual representation units in the target region and the visual representation units in the global context region are compressed to obtain the initial visual representation information.

[0454] The compression process includes:

[0455] Filtering operations for region units, and / or summarizing operations for region units;

[0456] The filtering operation for region units includes: filtering operations performed according to a predetermined region unit retention strategy and the target task requirement type of the image analysis task. The predetermined region unit retention strategy includes: the mapping relationship between each task requirement type and the region retention rule.

[0457] The summary generation operation for the region unit includes at least one of the following operations:

[0458] For any region unit, select a portion of the visual representation units within that region unit;

[0459] For any region unit, the visual representation units within that region unit are pooled.

[0460] For any region unit, the visual representation units within that region unit are mapped to a predetermined number of summary contents.

[0461] Optionally, the information construction unit compresses the visual representation units in the obtained target region unit and the visual representation units in the global context region to obtain initial visual representation information, including:

[0462] Based on the first lexical budget and the second lexical budget, the visual representation units in the obtained target region units and the visual representation units in the global context region are compressed to obtain the initial visual representation information.

[0463] Wherein, the first word element budget and the second word element budget are obtained by allocating the set total word element budget according to the first allocation rule and / or the second allocation rule;

[0464] The first allocation rule is: the lexical budget for the target region unit is higher than the lexical budget for the global context region;

[0465] The second allocation rule is as follows: when the target task requirement type of the image analysis task is the first type of requirement, the lexical budget for the target region unit is higher than the lexical budget for the global context region; when the target task requirement type of the image analysis task is the second type of requirement, the lexical budget for the global context region is higher than the lexical budget for the target region unit.

[0466] The first type of requirement is one that focuses on local details, while the second type of requirement is one that focuses on global semantics.

[0467] Optionally, the analysis module predicts whether the inference result of the target time step is a risky result based on the intermediate inference information of the target time step, including:

[0468] Based on the intermediate inference information of the target time step, at least one evaluation value is determined; wherein different evaluation values ​​are used to predict the unreliability of the inference result of the target time step from different prediction dimensions.

[0469] Based on the at least one evaluation value, predict the answer risk index of the reasoning result for the target time step;

[0470] If the risk index of the answer is higher than the predetermined risk threshold, the reasoning result of the target time step is determined to be a risky result.

[0471] Optionally, the intermediate inference information for the target time step includes the output probability of each candidate lexical unit;

[0472] The determination of at least one evaluation value based on the intermediate inference information of the target time step includes:

[0473] Based on the intermediate inference information of the target time step and the target task requirement type of the image analysis task, at least one evaluation value is determined;

[0474] Wherein, the at least one evaluation value includes:

[0475] At least one of the following: first evaluation value, second evaluation value, third evaluation value, fourth evaluation value, and fifth evaluation value;

[0476] Wherein, the first evaluation value represents the entropy of the probability distribution of the output probability of each candidate word;

[0477] The second evaluation value represents the difference between the highest output probability and the second highest output probability among the output probabilities of each candidate word.

[0478] The third evaluation value represents the number of output probabilities that are greater than a preset probability threshold among the output probabilities of each candidate word.

[0479] The fourth evaluation value represents the highest output probability among the output probabilities of each candidate word;

[0480] The fifth evaluation value represents the prior risk weight corresponding to the target task requirement type of the image analysis task.

[0481] Optionally, the determining module is specifically used for:

[0482] In the absence of a cached set of visual representations of the target image indicated by the image analysis task, the set of visual representations is constructed based on the target image.

[0483] If the visual representation set is cached, obtain the cached visual representation set;

[0484] The device further includes:

[0485] A caching module is used to cache the visual representation set after it has been constructed;

[0486] The analysis module performs supplementary visual context processing, including:

[0487] Based on the retrieved information, a subset of visual representation units are selected from the cached set of visual representations;

[0488] Based on the selected visual representation units, target supplementary information for the image analysis task is generated;

[0489] The target supplementary information is added to the current visual context;

[0490] The retrieval information includes:

[0491] The image analysis task, the target feature information, the currently generated inference result sequence of the large language model, the target task requirement type of the image analysis task, and the answer risk indicator are at least one of the following: the answer risk indicator is used to analyze whether a result is risky.

[0492] Optionally, the analysis module, based on retrieval information, filters a portion of the visual representation units from the cached visual representation set, including:

[0493] Based on the retrieved information, a region unit is selected from the region set containing multiple region units corresponding to the cached visual representation set to obtain a supplementary unit; wherein, each region unit contains a visual representation unit in the visual representation set whose corresponding sub-region belongs to the same local region of the image.

[0494] From the cached set of visual representations, obtain the visual representation units in the supplementary unit, and filter to obtain a portion of the visual representation units.

[0495] Optionally, each region unit is provided with metadata, and the metadata represents: the region type of the local image region to which the visual representation unit within the region unit belongs; the target feature information includes the target entity category indicated by the image analysis task;

[0496] The analysis module, based on the retrieved information, selects region units from the region set containing multiple region units corresponding to the cached visual representation set, to obtain supplementary verification units, including:

[0497] Based on the target entity category, a first region set is obtained by selecting region units from the region set containing multiple region units corresponding to the cached visual representation set.

[0498] Based on the target task requirement type of the image analysis task and the correspondence between each task requirement type and metadata, region units are selected from the obtained first region set to obtain a second region set; wherein, the metadata corresponding to each task requirement type is: the metadata required for the region unit to be selected under that task requirement type.

[0499] Select a supplementary proof unit from the second set of regions.

[0500] Optionally, the content in the current visual context can be cached as a key-value pair.

[0501] The analysis module adds the target supplementary information into the current visual context, including:

[0502] The information content of the target supplementary information is added to the current visual context in the form of key-value pairs;

[0503] or,

[0504] From the current visual context, identify content that meets the predetermined removal conditions, and replace the identified content with the information content of the target supplementary information in the form of key values.

[0505] Optionally, the analysis module performs supplementary processing of the visual context, including:

[0506] Before the inference results for this target time step are generated, supplementary processing of the visual context is performed;

[0507] Accordingly, the method further includes:

[0508] Trigger the backoff operation of the large language model for the target time step;

[0509] or,

[0510] The analysis module performs supplementary processing of the visual context, including:

[0511] Before the specified time step for startup is reached, perform supplementary processing of the visual context;

[0512] The specified time step is either the next time step after the target time step, or the time step used to generate the specified answer segment for the image analysis task.

[0513] Corresponding to the above method embodiments, this application also provides an electronic device, such as... Figure 8 As shown, it includes:

[0514] Memory 801 is used to store computer programs;

[0515] The processor 802 is used to implement a response method for image analysis tasks when executing programs stored in the memory 801.

[0516] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 802, the communication interface, and the memory 801 communicating with each other via the communication bus.

[0517] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0518] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0519] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0520] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0521] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements a response method for any of the above-described image analysis tasks.

[0522] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform a response to any of the image analysis tasks described in the above embodiments.

[0523] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0524] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0525] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0526] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A response method for an image analysis task, characterized in that, include: Obtain target feature information for image analysis tasks; Determine the set of visual representations of the target image indicated by the image analysis task; Based on the target feature information and the visual representation set, initial visual representation information for the image analysis task is generated; wherein, the initial visual representation information is used to characterize: Visual information of a portion of the visual representation units in the visual representation set; Using the initial visual representation information as the initial visual context, a large language model is invoked to process the image analysis task; Furthermore, after the large language model generates intermediate inference information for the target time step, it predicts whether the inference result for the target time step belongs to a risky result based on the intermediate inference information for the target time step. If it does, it performs supplementary processing of the visual context.

2. The method according to claim 1, characterized in that, The step of generating initial visual representation information for the image analysis task based on the target feature information and the visual representation set includes: Determine a region set corresponding to the visual representation set, which contains multiple region units; wherein each region unit contains a visual representation unit in the visual representation set whose corresponding sub-region belongs to the same local region of the image; Based on the target feature information, region units are selected from the region set to obtain target region units; Based on the visual representation units in the obtained target region units, the initial visual representation information for the image analysis task is constructed.

3. The method according to claim 2, characterized in that, The target feature information includes: the target entity category indicated by the image analysis task; The step of selecting target region units from the region set based on the target feature information to obtain target region units includes: Determine the entity category label for each region unit; wherein, the entity category label for each region unit is used to characterize: the entity category of the local image region to which the visual representation unit within that region unit belongs; Based on the entity category label of each region unit, a target region unit whose entity category label matches the target entity category is selected from the region set.

4. The method according to claim 3, characterized in that, Each region unit is provided with metadata, and the metadata represents the region type of the local image region to which the visual representation unit within that region unit belongs; Before constructing the initial visual representation information for the image analysis task based on the visual representation units in the obtained target region units, the method further includes: Based on the metadata of each region unit, region units with specified metadata are selected from the region set to obtain the global context region; wherein, the region type represented by the specified metadata is a region type that can represent the scene semantics of the target image; The initial visual representation information for the image analysis task, constructed based on the visual representation units in the obtained target region units, includes: Based on the visual representation units in the obtained target region units and the visual representation units in the global context region, the initial visual representation information for the image analysis task is constructed.

5. The method according to claim 4, characterized in that, The initial visual representation information for the image analysis task is constructed based on the visual representation units in the obtained target region units and the visual representation units in the global context region, including: The visual representation units in the target region and the visual representation units in the global context region are compressed to obtain the initial visual representation information. The compression process includes: Filtering operations for region units, and / or summarizing operations for region units; The filtering operation for region units includes: filtering operations performed according to a predetermined region unit retention strategy and the target task requirement type of the image analysis task. The predetermined region unit retention strategy includes: the mapping relationship between each task requirement type and the region retention rule. The summary generation operation for the region unit includes at least one of the following operations: For any region unit, select a portion of the visual representation units within that region unit; For any region unit, the visual representation units within that region unit are pooled. For any given region unit, the visual representation units within that region unit are mapped to a predetermined number of summary contents.

6. The method according to claim 5, characterized in that, The compression process for the visual representation units in the obtained target region unit and the visual representation units in the global context region to obtain initial visual representation information includes: Based on the first word budget and the second word budget, the visual representation units in the obtained target region units and the visual representation units in the global context region are compressed to obtain the initial visual representation information. Wherein, the first word element budget and the second word element budget are obtained by allocating the set total word element budget according to the first allocation rule and / or the second allocation rule; The first allocation rule is: the lexical budget for the target region unit is higher than the lexical budget for the global context region; The second allocation rule is as follows: when the target task requirement type of the image analysis task is the first type of requirement, the lexical budget for the target region unit is higher than the lexical budget for the global context region; when the target task requirement type of the image analysis task is the second type of requirement, the lexical budget for the global context region is higher than the lexical budget for the target region unit. The first type of requirement is one that focuses on local details, while the second type of requirement is one that focuses on global semantics.

7. The method according to any one of claims 1-6, characterized in that, The prediction of whether the inference result at the target time step is a risky result based on the intermediate inference information at that target time step includes: Based on the intermediate inference information of the target time step, at least one evaluation value is determined; wherein different evaluation values ​​are used to predict the unreliability of the inference result of the target time step from different prediction dimensions. Based on the at least one evaluation value, predict the answer risk index of the reasoning result for the target time step; If the risk index of the answer is higher than the predetermined risk threshold, the reasoning result of the target time step is determined to be a risky result.

8. The method according to claim 7, characterized in that, The intermediate inference information for this target time step includes the output probability of each candidate word; The determination of at least one evaluation value based on the intermediate inference information of the target time step includes: Based on the intermediate inference information of the target time step and the target task requirement type of the image analysis task, at least one evaluation value is determined; Wherein, the at least one evaluation value includes: At least one of the following: first evaluation value, second evaluation value, third evaluation value, fourth evaluation value, and fifth evaluation value; Wherein, the first evaluation value represents the entropy of the probability distribution of the output probability of each candidate word; The second evaluation value represents the difference between the highest output probability and the second highest output probability among the output probabilities of each candidate word. The third evaluation value represents the number of output probabilities that are greater than a preset probability threshold among the output probabilities of each candidate word. The fourth evaluation value represents the highest output probability among the output probabilities of each candidate word; The fifth evaluation value represents the prior risk weight corresponding to the target task requirement type of the image analysis task.

9. The method according to any one of claims 1-6, characterized in that, Determining the set of visual representations of the target image indicated by the image analysis task includes: In the absence of a cached set of visual representations of the target image indicated by the image analysis task, the set of visual representations is constructed based on the target image. If the visual representation set is cached, obtain the cached visual representation set; The method further includes: After constructing the visual representation set, the visual representation set is cached; The supplementary processing of the execution visual context includes: Based on the retrieved information, a subset of visual representation units are selected from the cached set of visual representations; Based on the selected visual representation units, target supplementary information for the image analysis task is generated; The target supplementary information is added to the current visual context; The retrieval information includes: The image analysis task, the target feature information, the currently generated inference result sequence of the large language model, the target task requirement type of the image analysis task, and the answer risk indicator are at least one of the following: the answer risk indicator is used to analyze whether a result is risky.

10. The method according to claim 9, characterized in that, The step of selecting a portion of visual representation units from the cached visual representation set based on retrieval information includes: Based on the retrieved information, a region unit is selected from the region set containing multiple region units corresponding to the cached visual representation set to obtain a supplementary unit; wherein, each region unit contains a visual representation unit in the visual representation set whose corresponding sub-region belongs to the same local region of the image. From the cached set of visual representations, obtain the visual representation units in the supplementary unit to filter and obtain a portion of the visual representation units.

11. The method according to claim 10, characterized in that, Each region unit is provided with metadata, and the metadata represents: the region type of the local image region to which the visual representation unit within the region unit belongs; the target feature information includes the target entity category indicated by the image analysis task; The step of selecting a region unit from a region set containing multiple region units corresponding to the cached visual representation set based on retrieval information to obtain a supplementary verification unit includes: Based on the target entity category, a first region set is obtained by selecting region units from the region set containing multiple region units corresponding to the cached visual representation set. Based on the target task requirement type of the image analysis task and the correspondence between each task requirement type and metadata, region units are selected from the obtained first region set to obtain a second region set; wherein, the metadata corresponding to each task requirement type is: the metadata required for the region unit to be selected under that task requirement type. Select a supplementary proof unit from the second set of regions.

12. The method according to claim 9, characterized in that, The content in the current visual context is cached in key-value format; The step of adding the target supplementary information into the current visual context includes: The information content of the target supplementary information is added to the current visual context in the form of key-value pairs; or, From the current visual context, identify content that meets the predetermined removal conditions, and replace the identified content with the information content of the target supplementary information in the form of key values.

13. The method according to any one of claims 1-6, characterized in that, The supplementary processing of the execution visual context includes: Before the inference results for this target time step are generated, supplementary processing of the visual context is performed; Accordingly, the method further includes: Trigger the backoff operation of the large language model for the target time step; or, The supplementary processing of the visual context includes: Before the specified time step for startup is reached, perform supplementary processing of the visual context; The specified time step is either the next time step after the target time step, or the time step used to generate the specified answer segment for the image analysis task.

14. A response device for an image analysis task, characterized in that, include: The acquisition module is used to obtain target feature information for image analysis tasks. The determination module is used to determine the set of visual representations of the target image indicated by the image analysis task; The generation module is used to generate initial visual representation information for the image analysis task based on the target feature information and the visual representation set; wherein, the initial visual representation information is used to characterize: the visual information of a portion of the visual representation units in the visual representation set; The calling module is used to call the large language model to process the image analysis task, using the initial visual representation information as the initial visual context. The analysis module is used to predict whether the inference result of the target time step is a risky result based on the intermediate inference information generated by the large language model at the target time step. If it is, supplementary processing of the visual context is performed.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-13.

Images