An image understanding method, apparatus, device, and medium

Abstract

This invention relates to the field of artificial intelligence technology, with applications in smart healthcare and finance. It discloses an image understanding method, apparatus, device, and medium. The method includes: acquiring an initial image and a question; decomposing the question into multiple fine-grained visual queries; constructing a relevance distribution of the initial image to the question using a visual-text alignment function; obtaining a visually relevant region map of the question; generating a perturbation image based on visual regions highly relevant to the question; generating an initial answer and a perturbation answer based on the initial image and question, and based on the perturbation image and question respectively; calculating the answer difference; obtaining visual utilization rate; when the visual utilization rate is lower than a preset threshold, calling at least two types of visual experts to process the image; superimposing the processing results onto the image to obtain an enhanced image; obtaining the enhanced visual utilization rate and the increase in visual utilization rate; selecting the visual expert with the largest increase in visual utilization rate as the optimal visual expert; and generating an answer based on the question and the enhanced image of the optimal visual expert.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and visual technology, and in particular to an image understanding method, apparatus, device, and medium. Background Technology

[0002] With the development of artificial intelligence and computer vision technologies, Large Vision-Language Models (LVLMs) have been widely used in tasks such as multimodal question answering, reasoning, and content generation. LVLMs achieve image understanding by fusing visual and linguistic information. However, existing image understanding using LVLMs typically generates answers based on linguistic patterns rather than visual features in the image, resulting in an overemphasis on language priors and lower accuracy and reliability of the reasoning results. Specifically, in the field of smart healthcare, failure to perform accurate and reliable image understanding analysis on user-uploaded image reports may affect the accuracy of medical diagnostic analysis. Similarly, in the financial sector, failure to perform accurate and reliable image understanding analysis on scanned images containing user identity information may lead to errors in information verification. Summary of the Invention

[0003] This invention provides an image understanding method, apparatus, device, and medium to solve the technical problem of low image understanding accuracy caused by generating answers based on language patterns rather than understanding images based on visual features in the image, thereby improving the accuracy and reliability of reasoning results.

[0004] Firstly, an image understanding method is provided, including: Obtain the initial image and question input by the user, decompose and rewrite the question into multiple fine-grained visual queries, construct the relevance distribution of the initial image to the question using a visual text alignment function, and obtain a visual relevance region map of the question; An initial answer is generated based on the initial image and the question. Then, based on a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map are masked or scrambled in the initial image to generate a perturbation image. A perturbation answer is generated based on the perturbation image and the question. The answer difference between the initial answer and the perturbation answer is calculated by a semantic encoder to obtain the visual utilization rate. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are invoked to process the initial image to obtain the corresponding expert structured visual features. The structured visual features of each expert are superimposed on the initial image using a visualization rendering function to obtain the corresponding enhanced image. The enhanced visual utilization rate of the corresponding visual expert is obtained based on the enhanced visual utilization rate of the visual expert. The visual utilization rate improvement of each visual expert is calculated based on the enhanced visual utilization rate of the visual expert and the visual utilization rate. The visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert. Obtain the enhanced image corresponding to the best visual expert, generate an answer based on the question and the enhanced image corresponding to the best visual expert, and use the answer as the final image understanding reasoning result.

[0005] Secondly, an image understanding device is provided, comprising: The image-question relevance module is used to obtain the initial image and question input by the user, decompose and rewrite the question into multiple fine-grained visual queries, construct the relevance distribution of the initial image to the question using a visual text alignment function, and obtain a visual relevance region map of the question. The visual utilization quantification module is used to generate an initial answer based on the initial image and the question, and, in conjunction with a preset perturbation rule, to mask or scramble visual regions with high relevance to the question in the initial image based on the visual relevance region map of the question, generate a perturbation image, generate a perturbation answer based on the perturbation image and the question, and calculate the answer difference between the initial answer and the perturbation answer through a semantic encoder to obtain the visual utilization rate. The optimal visual expert selection module is used to call at least two types of visual experts to process the initial image when the visual utilization rate is lower than a preset utilization rate threshold, obtain the corresponding expert structured visual features, and use a visualization rendering function to superimpose each expert structured visual feature onto the initial image to obtain the corresponding enhanced image. Based on each enhanced image, the enhanced visual utilization rate of the corresponding visual expert is obtained. Based on the enhanced visual utilization rate of the visual expert and the visual utilization rate, the visual utilization rate improvement of each visual expert is calculated, and the visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert. The reasoning result generation module is used to obtain the enhanced image corresponding to the best visual expert, generate an answer based on the question and the enhanced image corresponding to the best visual expert, and use the answer as the final image understanding reasoning result.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described image understanding method.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described image understanding method.

[0008] In the above-described image understanding method, apparatus, device, and medium, an initial image and question input by the user can be obtained through a client. The question is decomposed and rewritten into multiple fine-grained visual queries. A visual text alignment function is used to construct the relevance distribution of the initial image to the question, obtaining a visual relevance region map of the question. An initial answer is generated based on the initial image and the question. Combined with a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map of the question are masked or scrambled in the initial image to generate a perturbation image. A perturbation answer is generated based on the perturbation image and the question. The answer is then calculated using a semantic encoder. The visual utilization rate is obtained by comparing the answer difference between the initial answer and the perturbation answer. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are called to process the initial image to obtain the corresponding expert structured visual features. The structured visual features of each expert are superimposed on the initial image using a visualization rendering function to obtain the corresponding enhanced image. The enhanced visual utilization rate of the corresponding visual expert is obtained based on each enhanced image. The visual utilization rate improvement of each visual expert is calculated based on the enhanced visual utilization rate of the visual expert and the visual utilization rate. The visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert.The process involves obtaining an enhanced image corresponding to the optimal visual expert, generating an answer based on the question and the enhanced image, and feeding this answer back to the client as the final image understanding reasoning result. In this invention, for intelligent medical assistants or intelligent customer service in the smart healthcare field, or intelligent customer service in the financial field, an image understanding scheme can be used to obtain the initial image and question input by the user. The question is decomposed and rewritten into multiple fine-grained visual queries. A visual text alignment function is used to construct the relevance distribution of the initial image to the question, obtaining a visual relevance region map of the question. Combined with a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map are masked or scrambled in the initial image to generate a perturbation image. An initial answer and a perturbation answer are generated based on the initial image and the question, and based on the perturbation image and the question, respectively. The answer difference between the initial answer and the perturbation answer is calculated using a semantic encoder to obtain the visual utilization rate. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are invoked to process the image, obtaining the corresponding expert structured visual features. Visualization rendering functions are then used to render each expert structured visual feature... Visual features are superimposed onto the initial image to obtain corresponding enhanced images. The enhanced visual utilization rate of each visual expert is obtained based on these enhanced images. The improvement in visual utilization rate for each visual expert is calculated based on the enhanced visual utilization rate and the overall visual utilization rate. The visual expert with the largest improvement in visual utilization rate is selected as the optimal visual expert. The enhanced image corresponding to the optimal visual expert is obtained. An answer is generated based on the question and the enhanced image corresponding to the optimal visual expert. This answer serves as the final image understanding reasoning result. Visual utilization rate is used to accurately quantify the dependence of visual features on the initial image, providing an objective basis for subsequent use of visual experts. This approach, based on visual utilization rate, automatically selects the optimal visual expert after detecting low visual feature utilization. Visual features are enhanced using the optimal visual expert, and reasoning is performed using the enhanced images. This increases the dependence on visual features during image understanding, making the reasoning result more correlated with the initial image. Furthermore, it improves the model's focus on key visual regions without retraining large-scale visual language models, reducing the risk of hallucinations and improving the accuracy and reliability of reasoning results. It is also adaptable to different types of visual experts, exhibiting strong versatility and scalability. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for the image understanding method in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an image understanding method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an image understanding device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The image understanding method provided in this embodiment of the invention can be applied to, for example... Figure 1In application environments, intelligent assistants or intelligent customer service used in smart healthcare and finance are usually implemented through a server, where the client communicates with the server via a network. The server can obtain the initial image and question input by the user from the client, decompose and rewrite the question into multiple fine-grained visual queries, construct the relevance distribution of the initial image to the question using a visual text alignment function, and obtain a visual relevance region map of the question. An initial answer is generated based on the initial image and the question. Using a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map of the question are masked or scrambled in the initial image to generate a perturbation image. A perturbation answer is generated based on the perturbation image and the question. The answer difference between the initial answer and the perturbation answer is calculated using a semantic encoder to obtain the visual utilization rate. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are invoked to process the initial image, obtaining the corresponding expert structured visual features. Each expert structured visual feature is superimposed onto the initial image using a visualization rendering function to obtain the corresponding enhanced image. The enhanced visual utilization rate of the corresponding visual expert is obtained based on each enhanced image. The visual utilization rate improvement of each visual expert is calculated based on the enhanced visual utilization rate and the overall visual utilization rate. The visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert.The process involves obtaining an enhanced image corresponding to the optimal visual expert, generating an answer based on the question and the enhanced image, and feeding this answer back to the client as the final image understanding reasoning result. In this invention, for intelligent medical assistants or intelligent customer service in the smart healthcare field, or intelligent customer service in the financial field, an image understanding scheme can be used to obtain the initial image and question input by the user. The question is decomposed and rewritten into multiple fine-grained visual queries. A visual text alignment function is used to construct the relevance distribution of the initial image to the question, obtaining a visual relevance region map of the question. Combined with a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map are masked or scrambled in the initial image to generate a perturbation image. An initial answer and a perturbation answer are generated based on the initial image and the question, and based on the perturbation image and the question, respectively. The answer difference between the initial answer and the perturbation answer is calculated using a semantic encoder to obtain the visual utilization rate. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are invoked to process the initial image, obtaining the corresponding expert structured visual features. Visualization rendering functions are then used to structure each expert's visual features. Visual features are superimposed onto the initial image to obtain corresponding enhanced images. The enhanced visual utilization rate of each visual expert is obtained based on these enhanced images. The visual utilization rate improvement of each visual expert is calculated based on the enhanced visual utilization rate and the visual utilization rate of the expert with the largest improvement. The enhanced image corresponding to the optimal visual expert is obtained. An answer is generated based on the question and the enhanced image corresponding to the optimal visual expert. This answer serves as the final image understanding reasoning result. Visual utilization rate is used to accurately quantify the dependence of the initial image's visual features, providing an objective basis for subsequent use of visual experts. This approach, based on visual utilization rate, automatically selects the optimal visual expert after detecting low visual feature utilization. Visual features are enhanced using the optimal visual expert, and reasoning is performed using the enhanced images. This increases the dependence on visual features during image understanding, making the reasoning result more relevant to the initial image. Furthermore, it improves the model's focus on key visual regions without retraining large-scale visual language models, reducing the risk of hallucinations and improving the accuracy and reliability of reasoning results. It is also adaptable to different types of visual experts, exhibiting strong versatility and scalability. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0013] Please see Figure 2 As shown, Figure 2A flowchart illustrating the image understanding method provided in this embodiment of the invention includes the following steps: S10: Obtain the initial image and question input by the user, decompose and rewrite the question into multiple fine-grained visual queries, construct the relevance distribution of the initial image to the question using a visual text alignment function, and obtain a visual relevance region map of the question.

[0014] Fine-grained visual queries refer to queries on the visual features of the initial image that are relevant to the question. The question can be decomposed and rewritten into multiple fine-grained visual queries using a language model, transforming the vague and abstract question into fine-grained queries that can be understood by the visual model. Each fine-grained visual query corresponds to a specific and single visual feature acquisition action. The results of all fine-grained visual queries can be concatenated to obtain a complete answer, which can be used to answer the input question. The visual-text alignment function is a similarity calculation function based on a cross-modal attention mechanism. It is used to quantify the semantic relevance of each pixel or region in the initial image to the sub-questions corresponding to the fine-grained visual queries. The visual-text alignment function is used to construct the relevance distribution of the initial image to the question, reflecting the proportion of relevance between the initial image and the question. This reveals which regions of the initial image have a higher matching degree with the question, and a visual relevance map of the question can be obtained based on the relevance distribution of the initial image to the question, facilitating subsequent image understanding. The visual relevance map of the question reflects the relevance distribution of the question in the initial image. The image understanding method provided by this invention can be applied to intelligent assistants or intelligent customer service in various application scenarios such as healthcare and finance. Intelligent assistants or intelligent customer service are typically implemented through a server that can receive initial input images and questions in real time. For example, in the medical field, users often use intelligent medical assistants or intelligent customer service installed on terminals such as computers or mobile phones for medical consultations. These assistants or customer service can interpret and analyze physical examination reports, imaging reports, medical images, or screenshots from home monitoring devices to understand the reports or perform diagnostic analysis on the monitoring results. For instance, the initial input image can be a screenshot of a physical examination report, an imaging report, a medical image, or a screenshot from a monitoring device. In medical application scenarios, the object in the medical image is classified as a lesion, i.e., a part of the body where a disease has occurred. Medical imaging refers to images of internal tissues obtained non-invasively for medical treatment or research, such as images of the stomach, abdomen, heart, knees, and brain. Examples include CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound (US), X-ray images, electroencephalograms (EEGs), and images generated by medical instruments such as optical imaging. Monitoring devices include blood glucose meters and blood pressure monitors. When the initial input image is a screenshot of a blood pressure monitor, the input question might be: "Is the blood pressure value within the normal range?". In this case, a language model can break down and rewrite the question into multiple fine-grained visual queries, including: "What is the unit of the blood pressure value in the image?", "What is the systolic blood pressure value shown in the image?", "What is the diastolic blood pressure value shown in the image?", and "Is the normal reference range for blood pressure marked in the image?".

[0015] Alternatively, in the financial sector, users can consult with intelligent financial assistants or customer service representatives installed on computers or mobile phones. These assistants or representatives can verify and analyze images of signed documents or reports to facilitate information verification, document analysis, and transaction consultation. For instance, when the initial input image is a scanned image of a signed document from a transaction process, the input question could be: "Does the signed document have legal validity?" A language model can then break down and rewrite the question into multiple fine-grained visual queries, including: "Is the signatory's name in the image the designated contact person for this financial transaction?", "Is the signing date in the image within the valid processing period of this financial transaction?", "Does the identification label next to the signatory in the image match the signing identity required for this financial transaction?", "Does the image contain a compliant handwritten signature or an officially recognized electronic seal?", "Does a single page image of a multi-page document have complete seal marks across the binding?", and "Does the image contain clauses related to validity, and have these clauses not been altered?"

[0016] Preferably, in some embodiments, the step S10 of decomposing and rewriting the problem into multiple fine-grained visual queries includes: The semantics of the question are analyzed to identify the multi-dimensional elements of the answer; among these, the semantics of the question can be analyzed through a language model. Based on the multi-dimensional answer elements of the question and the visual features of the initial image, the question is decomposed and rewritten into multiple fine-grained visual queries, resulting in a set of fine-grained visual queries. Each fine-grained visual query may include the visual features of the initial image and one dimension of the question's answer elements.

[0017] Preferably, in some embodiments, step S10, which involves constructing the relevance distribution of the initial image to the problem using a visual text alignment function to obtain a visually relevant region map of the problem, includes: The image encoder using the CLIP model extracts the visual feature vector of the initial image, and the text encoder using the CLIP model converts each fine-grained visual query into a corresponding text feature vector. The CLIP (Contrastive Language-Image Pre-training) model is a multimodal pre-training model that achieves semantic alignment between images and text through contrastive learning. The CLIP model is used to calculate the cosine similarity between the image feature vector and the text feature vector, and the mapping distribution area of ​​the pixel block corresponding to each cosine similarity in the initial image is obtained. Determine whether each cosine similarity is lower than a preset similarity threshold. Based on the determination result, combine the binarization processing method to process the pixels in the mapping distribution area of ​​the pixel block corresponding to each cosine similarity in the initial image to obtain the problem visual relevance region map.

[0018] Here, the cosine similarity between the image feature vector and the text feature vector represents the degree of matching between the pixel block corresponding to each image feature vector and the fine-grained visual query. It represents the relevance distribution of the corresponding fine-grained visual query in the initial image, thus obtaining the relevance distribution of the question in the initial image. The pixel block corresponding to each cosine similarity is the pixel block corresponding to the cosine similarity between each image feature vector and the text feature vector. The mapping distribution region of the pixel block in the initial image refers to the corresponding region of the initial image where the pixel block is mapped. The CLIP model ensures that the obtained visual feature vector and text feature vector have equal dimensions, facilitating the calculation of the correlation between the text feature vector and the visual feature vector.

[0019] Specifically, the step of processing the pixels in the mapping distribution region of the initial image corresponding to each cosine similarity based on the judgment result and the binarization processing method includes: Pixels in the mapping distribution area of ​​the initial image whose cosine similarity is not lower than a preset similarity threshold are marked as 1; Pixels in the mapping distribution area of ​​the initial image whose cosine similarity is lower than a preset similarity threshold are marked as 0.

[0020] In the visual relevance map of the problem, the mapping distribution area marked with a pixel of 1 is a visual area with high relevance to the problem, and the mapping distribution area marked with a pixel of 0 is a visual area with low relevance to the problem.

[0021] Specifically, in some embodiments, before the image encoder using the CLIP model extracts the visual feature vector of the initial image, and before the text encoder using the CLIP model converts each fine-grained visual query into a corresponding text feature vector, the image understanding method further includes: The initial image and each fine-grained visual query undergo format conversion, preserving the original pixel size of the initial image. This ensures that the converted initial image is in a format processable by the CLIP model's image encoder, and the converted fine-grained visual queries are in a format processable by the CLIP model's text encoder. The initial image's format conversion can be achieved by scaling the image and normalizing pixel values, resulting in a 224×224 pixel size. Adding fixed prefixes and / or suffixes can make the fine-grained visual queries conform to the input requirements of the CLIP model's text encoder. Preserving the original pixel size of the initial image facilitates subsequent pixel-level mapping, allowing for the mapping distribution of pixel blocks corresponding to each cosine similarity score within the initial image's mapping distribution area. Since the pixel size of the problem visual relevance region map is equal to the original pixel size of the initial image, each pixel position in the problem visual relevance region map corresponds one-to-one with each pixel position in the initial image.

[0022] S20: Generate an initial answer based on the initial image and the question. Combined with a preset perturbation rule, mask or scramble the visual regions in the initial image that are highly relevant to the question based on the visual relevance region map of the question to generate a perturbation image. Generate a perturbation answer based on the perturbation image and the question. Calculate the answer difference between the initial answer and the perturbation answer using a semantic encoder to obtain the visual utilization rate. This process employs a large-scale visual language model to generate initial answers based on an initial image and a question. It then maps regions in the initial image that are highly relevant to the question's visual relevance to the question's visual region map to perform perturbation operations. This precisely perturbs highly relevant regions while retaining less relevant regions, facilitating the subsequent determination of the large-scale visual language model's utilization rate of the initial image's visual features during multimodal question-answering reasoning and preventing low-relevance regions from interfering with the calculation of visual utilization. Preset perturbation rules include different perturbation occurrence ratios and different perturbation location positions. Multiple perturbation occurrence ratios and locations allow for the generation of different perturbation images based on the corresponding ratios or locations. This enables the generation of corresponding perturbation images based on the question and the corresponding perturbation images. The preset perturbation rules control the perturbation intensity, preventing excessive perturbation that could distort the answer. Pixel sampling of the corresponding regions during perturbation is randomized to ensure the diversity of perturbation versions. By employing a large-scale visual language model to generate perturbation answers based on the perturbation image and the question, the visual utilization rate can be used to measure whether the large-scale visual language model uses question-related regions in the initial image, thus quantifying the dependence of the large-scale visual language model on the visual features of the initial image. Masking refers to randomly replacing pixels in the image with solid color blocks or random noise. The proportion of pixels replaced in a corresponding region can be any perturbation occurrence proportion in a preset perturbation rule. Therefore, combining the preset perturbation rule, masking the visual regions in the initial image that are highly relevant to the question from the question's visually relevant region map can specifically involve: randomly selecting pixels in the initial image with a perturbation occurrence proportion from the visual regions in the question's visually relevant region map that are highly relevant to the question, and replacing them. Alternatively, combining the preset perturbation rule, masking the visual regions in the initial image that are highly relevant to the question from the question's visually relevant region map can specifically involve: selecting pixels in the initial image from the corresponding region based on the location of any perturbation occurrence region from the visual regions in the question's visually relevant region map that are highly relevant to the question, and replacing them. Scrambling refers to disrupting the pixel order of an image to destroy its original visual semantics. Combining preset perturbation rules, scrambling the initial image based on the visual regions in the problem's visual relevance map that are highly relevant to the problem can be specifically performed as follows: randomly selecting a number of pixels in the initial image with a perturbation occurrence ratio based on the visual regions in the problem's visual relevance map that are highly relevant to the problem, and rearranging the pixels accordingly.Alternatively, by combining preset perturbation rules, the initial image can be scrambled based on the visual regions in the problem's visual correlation map that are highly relevant to the problem. Specifically, this can be done by selecting the corresponding region's pixels in the initial image and rearranging them based on the location of any perturbation region, according to the visual regions in the problem's visual correlation map that are highly relevant to the problem.

[0023] Preferably, step S20, which involves combining preset perturbation rules to mask or scramble visual regions in the initial image that are highly relevant to the problem according to the visual relevance region map, includes: By combining preset perturbation rules, the pixel values ​​of the corresponding regions in the initial image are replaced with preset fixed values ​​based on the visually relevant regions of the problem's visual region map that are highly relevant to the problem, thus obtaining a perturbation mask. Alternatively, by combining preset perturbation rules, the pixel values ​​of the corresponding regions in the initial image are randomly shuffled based on the visually relevant regions of the problem's visual region map that are highly relevant to the problem, thus obtaining a perturbation mask. The preset fixed values ​​can be black pixel values, white pixel values, or average pixel values. By replacing the pixel values ​​of the regions in the initial image that are highly relevant to the problem, the visual features of the corresponding regions are completely obscured. By randomly shuffling the pixel values ​​of the regions in the initial image that are highly relevant to the problem, the visual structure within the corresponding regions is disrupted, but the overall position and size of the regions are not changed. The perturbation mask is a grayscale image or binary image with the same original pixel size as the initial image. A pixel value of 1 in the perturbation mask represents that the corresponding pixel position remains the same as the pixel in the initial image, while a pixel value of 0 in the perturbation mask represents that the corresponding pixel position is perturbed.

[0024] Preferably, the perturbation image is obtained using formula (1): (1) In the formula, This represents the perturbation mask. Represents an image. This indicates a perturbed image. This indicates element-wise multiplication.

[0025] Specifically, the step of calculating the answer difference between the initial answer and the perturbation answer using a semantic encoder to obtain visual utilization includes: The initial answer is semantically encoded using a semantic encoder to obtain the initial answer encoding; The perturbation answer is semantically encoded using a semantic encoder to obtain the perturbation answer code; Calculate the similarity between the initial answer code and the perturbed answer code; The visual utilization rate is obtained by calculating the perturbation similarity expectation based on the similarity between the initial answer code and the perturbation answer code of the multiple perturbation images.

[0026] Preferably, the visual utilization rate is calculated using formula (2): (2) In the formula, This represents the initial answer. This indicates a perturbation of the answer. This represents the semantic encoder function. This represents the similarity calculation function. A function for calculating the expectation of the similarity of multiple perturbations. This represents the visual utilization rate. A higher similarity between the perturbation answer and the initial answer indicates a lower probability that the large-scale visual language model will utilize the visual features of the initial image for multimodal question-answering reasoning.

[0027] S30: When the visual utilization rate is lower than the preset utilization rate threshold, at least two types of visual experts are called to process the initial image to obtain the corresponding expert structured visual features. The visual rendering function is used to superimpose each expert structured visual feature onto the initial image to obtain the corresponding enhanced image. The enhanced visual utilization rate of the corresponding visual expert is obtained according to each enhanced image. The visual utilization rate improvement of each visual expert is calculated based on the enhanced visual utilization rate of the visual expert and the visual utilization rate. The visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert. In this context, visual experts refer to modular algorithms or models focused on solving specific types of visual feature extraction tasks. They accurately extract corresponding structured visual features from images, supplementing large-scale visual language models with key visual features and compensating for their limitations in extracting specific types of visual features. The output of a visual expert is expert-structured visual features. Each visual expert focuses on only one specific type of visual feature extraction task, and all visual expert outputs are standardized results. Visual experts include segmentation experts, detection experts, Optical Character Recognition (OCR) experts, and visual embedding-based detection experts. The segmentation expert performs semantic segmentation on images, outputting semantic masks with each pixel corresponding to a category label. This assists large-scale visual language models in accurately identifying specific category regions within an image, addressing the problem of indistinguishable object boundaries. The detection expert performs object detection on images, outputting bounding boxes and category labels. The bounding boxes contain the coordinates of each object, assisting large-scale visual language models in locating the position and quantity of key objects in an image, addressing the problem of inaccurate object localization. The optical character recognition expert performs optical character recognition and detection on images, outputting all recognizable text content and text positions within an image. The text content is a string, and the text positions are coordinates, assisting large-scale visual language models in reading text information from images, addressing the problem of large-scale visual language models struggling to directly recognize text in images. The visual embedding-based detection expert performs visual feature retrieval on images, outputting feature descriptions or reference images similar to the visual features of key regions in the image. This provides additional visual context support for large-scale visual language models, addressing the problem of insufficient visual features in images.

[0028] Preferably, the step of superimposing the expert structured visual features onto the initial image using a visualization rendering function to obtain the corresponding enhanced image includes: The expert structured visual features obtained after processing the initial image by different visual experts are superimposed onto the initial image using a visualization rendering function to obtain the enhanced image corresponding to each visual expert.

[0029] For segmentation experts, the expert structured visual features output after processing the initial input image can be semantic masks. These masks can be overlaid onto corresponding areas of the initial image using semi-transparent layers of different colors to obtain the enhanced image for segmentation. For detection experts, the expert structured visual features output after processing the initial input image can be bounding boxes and category labels. Bounding boxes with category labels can be used to label corresponding objects in the initial image to obtain the enhanced image for detection. For optical character recognition (OCR) experts, the expert structured visual features output after processing the initial input image can be text content and text location. The text content can be overlaid onto the corresponding text area of ​​the initial image as a floating label to obtain the enhanced image for OCR. For visual embedding-based detection experts, the expert structured visual features output after processing the initial input image are feature descriptions or reference images similar to the key region visual features of the initial image. These can be overlaid onto the corresponding areas of the initial image as thumbnails or feature heatmaps to obtain the enhanced image for visual embedding-based detection.

[0030] Specifically, step S30, which involves obtaining the enhanced visual utilization rate of the corresponding visual expert based on each enhanced image, includes the following steps: Based on preset perturbation rules, visual regions with high relevance to the problem in the visual relevance region map are masked or scrambled in the enhanced image to generate multiple enhanced perturbation masks. By employing the enhanced perturbation mask through element-wise multiplication, and based on the visual regions in the problem's visual relevance map that are highly relevant to the problem, the corresponding enhanced image is targeted and perturbed to obtain the corresponding enhanced perturbation image. An initial enhanced answer is generated based on the enhanced image and the question; wherein, a large-scale visual language model can be used to generate the initial enhanced answer based on the enhanced image and the question. An enhanced perturbation answer is generated based on the enhanced perturbation image and the question; wherein, the enhanced perturbation answer can be generated based on the enhanced perturbation image and the question by employing a large-scale visual language model; The initial enhanced answer is semantically encoded using a semantic encoder to obtain the initial enhanced answer code; The enhanced perturbation answer is semantically encoded using a semantic encoder to obtain the enhanced perturbation answer code; Calculate the similarity between the initial enhanced answer code and the enhanced perturbation answer code; The enhanced perturbation similarity expectation is calculated based on the similarity between the initial enhanced answer code and the enhanced perturbation answer code of the multiple enhanced perturbation images to obtain the enhanced visual utilization rate.

[0031] S40: Obtain the enhanced image corresponding to the optimal visual expert, and generate an answer based on the question and the enhanced image corresponding to the optimal visual expert. This answer serves as the final image understanding and reasoning result. Specifically, a large-scale visual language model can be used to generate the answer based on the question and the enhanced image corresponding to the optimal visual expert.

[0032] As can be seen, in the above solution, for intelligent medical assistants or intelligent customer service in the field of smart healthcare, or for intelligent customer service in the financial field, an image understanding approach can be used to obtain the initial image and question input by the user. The question is decomposed and rewritten into multiple fine-grained visual queries. A visual text alignment function is used to construct the relevance distribution of the initial image to the question, obtaining a visual relevance region map of the question. Combined with a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map of the question are masked or scrambled in the initial image to generate a perturbation image. An initial answer and a perturbation answer are generated based on the initial image and the question, and based on the perturbation image and the question, respectively. The answer difference between the initial answer and the perturbation answer is calculated by a semantic encoder to obtain the visual utilization rate. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are called to process the initial image to obtain the corresponding expert structured visual features. Each expert structured visual feature is superimposed onto the initial image using a visualization rendering function to obtain the corresponding enhanced image. The process involves acquiring the enhanced visual utilization rate of the corresponding visual expert for the image, calculating the improvement in visual utilization rate for each visual expert based on the enhanced visual utilization rate, selecting the visual expert with the largest improvement as the optimal visual expert, acquiring the enhanced image corresponding to the optimal visual expert, generating an answer based on the question and the enhanced image corresponding to the optimal visual expert, and using this answer as the final image understanding inference result. Visual utilization rate is used to accurately quantify the dependence of the initial image's visual features, providing an objective basis for subsequent use of visual experts. This process, centered on visual utilization rate, automatically selects the optimal visual expert after detecting low visual feature utilization, enhances visual features based on the optimal visual expert, and uses the enhanced image for inference, increasing the dependence on visual features during image understanding. This results in a higher correlation between the inference result and the initial image, and improves the model's attention to key visual regions without retraining a large-scale visual language model, reducing the risk of hallucinations and improving the accuracy and reliability of the inference result. Furthermore, it is adaptable to different types of visual experts, exhibiting strong versatility and scalability.

[0033] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0034] In one embodiment, an image understanding device is provided, which corresponds one-to-one with the image understanding methods described in the above embodiments. For example... Figure 3 As shown, the image understanding device includes an image problem relevance module 101, a visual utilization quantification module 102, an optimal visual expert selection module 103, and a reasoning result generation module 104. Detailed descriptions of each functional module are as follows: The image-question relevance module 101 is used to obtain the initial image and question input by the user, decompose and rewrite the question into multiple fine-grained visual queries, construct the relevance distribution of the initial image to the question using a visual text alignment function, and obtain a visual relevance region map of the question. The visual utilization quantification module 102 is used to generate an initial answer based on the initial image and the question, combine a preset perturbation rule, and block or scramble the visual regions with high relevance to the question in the initial image according to the visual relevance region map of the question to generate a perturbation image, generate a perturbation answer based on the perturbation image and the question, and calculate the answer difference between the initial answer and the perturbation answer through a semantic encoder to obtain the visual utilization rate; The optimal visual expert selection module 103 is used to call at least two types of visual experts to process the initial image when the visual utilization rate is lower than a preset utilization rate threshold, obtain the corresponding expert structured visual features, and use a visualization rendering function to superimpose each expert structured visual feature onto the initial image to obtain the corresponding enhanced image. Based on each enhanced image, the enhanced visual utilization rate of the corresponding visual expert is obtained. Based on the enhanced visual utilization rate of the visual expert and the visual utilization rate, the visual utilization rate improvement of each visual expert is calculated, and the visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert. The reasoning result generation module 104 is used to obtain the enhanced image corresponding to the best visual expert, generate an answer based on the question and the enhanced image corresponding to the best visual expert, and use the answer as the final image understanding reasoning result.

[0035] In one embodiment, the image problem correlation module 101 is specifically used for: Analyze the semantics of the question and identify the multi-dimensional elements of the answer; Based on the multi-dimensional answer elements of the question and the visual features of the initial image, the question is decomposed and rewritten into multiple fine-grained visual queries to obtain a set of fine-grained visual queries.

[0036] In one embodiment, the image problem correlation module 101 is specifically used for: An image encoder using the CLIP model extracts the visual feature vector of the initial image, and a text encoder using the CLIP model converts each fine-grained visual query into a corresponding text feature vector. The CLIP model is used to calculate the cosine similarity between the image feature vector and the text feature vector, and the mapping distribution area of ​​the pixel block corresponding to each cosine similarity in the initial image is obtained. Determine whether each cosine similarity is lower than a preset similarity threshold. Based on the determination result, combine the binarization processing method to process the pixels in the mapping distribution area of ​​the pixel block corresponding to each cosine similarity in the initial image to obtain the problem visual relevance region map.

[0037] In one embodiment, the image problem correlation module 101 is further configured to: Pixels in the mapping distribution area of ​​the initial image whose cosine similarity is not lower than a preset similarity threshold are marked as 1; Pixels in the mapping distribution area of ​​the initial image whose cosine similarity is lower than a preset similarity threshold are marked as 0.

[0038] In one embodiment, the visual utilization quantification module 102 is specifically used for: By combining preset perturbation rules, the pixel values ​​of the corresponding regions in the initial image are set to preset fixed values ​​based on the visual regions in the problem visual correlation map that are highly relevant to the problem, thereby obtaining a perturbation mask; or, by combining preset perturbation rules, the pixel values ​​of the corresponding regions in the initial image are randomly shuffled based on the visual regions in the problem visual correlation map that are highly relevant to the problem, thereby obtaining a perturbation mask.

[0039] In one embodiment, the optimal visual expert selection module 103 is specifically used for: Based on preset perturbation rules, visual regions with high relevance to the problem in the visual relevance region map are masked or scrambled in the enhanced image to generate multiple enhanced perturbation masks. By employing the enhanced perturbation mask through element-wise multiplication, and based on the visual regions in the problem's visual relevance map that are highly relevant to the problem, the corresponding enhanced image is targeted and perturbed to obtain the corresponding enhanced perturbation image. Generate an initial enhanced answer based on the enhanced image and the question; An enhanced perturbation answer is generated based on the enhanced perturbation image and the question; The initial enhanced answer is semantically encoded using a semantic encoder to obtain the initial enhanced answer code; The enhanced perturbation answer is semantically encoded using a semantic encoder to obtain the enhanced perturbation answer code; Calculate the similarity between the initial enhanced answer code and the enhanced perturbation answer code; The enhanced perturbation similarity expectation is calculated based on the similarity between the initial enhanced answer code and the enhanced perturbation answer code of the multiple enhanced perturbation images to obtain the enhanced visual utilization rate.

[0040] This invention provides an image understanding device. It acquires an initial image and a question input by a user through an image-question relevance module. The question is decomposed and rewritten into multiple fine-grained visual queries. A visual text alignment function is used to construct the relevance distribution of the initial image to the question, obtaining a visual relevance region map. A visual utilization quantification module, combined with preset perturbation rules, masks or scrambles visual regions in the initial image based on high relevance to the question in the visual relevance region map, generating a perturbation image. Initial and perturbation answers are generated based on the initial image and the question, and also based on the perturbation image and the question. A semantic encoder calculates the answer difference between the initial and perturbation answers to obtain visual utilization. An optimal visual expert selection module, when the visual utilization is lower than a preset utilization threshold, calls at least two types of visual experts to process the initial image, obtaining corresponding expert structured visual features. Each expert structured visual feature is superimposed onto the initial image using a visualization rendering function to obtain a corresponding enhanced image. A corresponding visual expert is then obtained based on each enhanced image. The system enhances the visual utilization rate of visual experts by calculating the improvement in visual utilization rate for each expert, based on the enhanced visual utilization rate and the overall visual utilization rate. The expert with the largest improvement is selected as the optimal visual expert. An enhanced image corresponding to the optimal visual expert is obtained through the inference result generation module. An answer is generated based on the question and the enhanced image corresponding to the optimal visual expert, serving as the final image understanding inference result. Visual utilization rate is used to accurately quantify the dependence of the initial image's visual features, providing an objective basis for subsequent use of visual experts. This system, centered on visual utilization rate, automatically selects the optimal visual expert when low visual feature utilization is detected. Visual features are enhanced based on the optimal visual expert, and inference is performed using the enhanced image. This increases the dependence on visual features during image understanding, making the inference result more relevant to the initial image. Furthermore, it improves the model's focus on key visual regions without requiring retraining of large-scale visual language models, reducing the risk of hallucinations and improving the accuracy and reliability of inference results. It is also adaptable to different types of visual experts, exhibiting strong versatility and scalability.

[0041] For specific limitations regarding the image understanding device, please refer to the limitations of the image understanding method above, which will not be repeated here. Each module in the aforementioned image understanding device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0042] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of an image understanding method on the server side.

[0043] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a client-side image understanding method.

[0044] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the initial image and question input by the user, decompose and rewrite the question into multiple fine-grained visual queries, construct the relevance distribution of the initial image to the question using a visual text alignment function, and obtain a visual relevance region map of the question; An initial answer is generated based on the initial image and the question. Then, based on a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map are masked or scrambled in the initial image to generate a perturbation image. A perturbation answer is generated based on the perturbation image and the question. The answer difference between the initial answer and the perturbation answer is calculated by a semantic encoder to obtain the visual utilization rate. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are invoked to process the initial image to obtain the corresponding expert structured visual features. The structured visual features of each expert are superimposed on the initial image using a visualization rendering function to obtain the corresponding enhanced image. The enhanced visual utilization rate of the corresponding visual expert is obtained based on the enhanced visual utilization rate of the visual expert. The visual utilization rate improvement of each visual expert is calculated based on the enhanced visual utilization rate of the visual expert and the visual utilization rate. The visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert. Obtain the enhanced image corresponding to the best visual expert, generate an answer based on the question and the enhanced image corresponding to the best visual expert, and use the answer as the final image understanding reasoning result.

[0045] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the initial image and question input by the user, decompose and rewrite the question into multiple fine-grained visual queries, construct the relevance distribution of the initial image to the question using a visual text alignment function, and obtain a visual relevance region map of the question; An initial answer is generated based on the initial image and the question. Then, based on a preset perturbation rule, visual regions with high relevance to the question in the visual relevance region map are masked or scrambled in the initial image to generate a perturbation image. A perturbation answer is generated based on the perturbation image and the question. The answer difference between the initial answer and the perturbation answer is calculated by a semantic encoder to obtain the visual utilization rate. When the visual utilization rate is lower than a preset utilization rate threshold, at least two types of visual experts are invoked to process the initial image to obtain the corresponding expert structured visual features. The structured visual features of each expert are superimposed on the initial image using a visualization rendering function to obtain the corresponding enhanced image. The enhanced visual utilization rate of the corresponding visual expert is obtained based on the enhanced visual utilization rate of the visual expert. The visual utilization rate improvement of each visual expert is calculated based on the enhanced visual utilization rate of the visual expert and the visual utilization rate. The visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert. Obtain the enhanced image corresponding to the best visual expert, generate an answer based on the question and the enhanced image corresponding to the best visual expert, and use the answer as the final image understanding reasoning result.

[0046] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0047] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0048] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0049] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image understanding method characterized by, The method comprises the following steps: acquiring an initial image and a question input by a user, decomposing and rewriting the question into a plurality of fine-grained visual queries, constructing a relevance distribution of the initial image to the question by using a visual text alignment function, and acquiring a question visual relevant region map; generating an initial answer according to the initial image and the question, combining a preset perturbation rule, performing shielding or scrambling on the initial image according to a visual region with high relevance to the question in the question visual relevant region map, generating a perturbation image, generating a perturbation answer according to the perturbation image and the question, calculating an answer difference between the initial answer and the perturbation answer by using a semantic encoder, and acquiring a visual utilization rate; when the visual utilization rate is lower than a preset utilization rate threshold, calling at least two types of visual experts to process the initial image, acquiring corresponding expert structured visual features, respectively superimposing each of the expert structured visual features to the initial image by using a visual rendering function to obtain a corresponding enhanced image, acquiring an enhanced visual utilization rate of the corresponding visual expert according to each of the enhanced images, calculating a visual utilization rate improvement amount of each visual expert according to the enhanced visual utilization rate of the visual expert and the visual utilization rate, and selecting a visual expert with the largest visual utilization rate improvement amount as an optimal visual expert; acquiring the enhanced image corresponding to the optimal visual expert, and generating an answer according to the question and the enhanced image corresponding to the optimal visual expert, the answer being taken as a final image understanding reasoning result.

2. The image understanding method of claim 1, wherein, The method further comprises the following steps: parsing semantics of the question and identifying multi-dimensional answer elements of the question; decomposing and rewriting the question into a plurality of fine-grained visual queries according to the multi-dimensional answer elements of the question and visual features of the initial image to obtain a fine-grained visual query set.

3. The image understanding method of claim 1, wherein, The method further comprises the following steps: extracting a visual feature vector of the initial image by using an image encoder of a CLIP model, converting each fine-grained visual query into a corresponding text feature vector by using a text encoder of the CLIP model; calculating cosine similarity between the image feature vector and the text feature vector by using the CLIP model, and acquiring a mapping distribution region of a pixel block corresponding to each cosine similarity in the initial image; determining whether each cosine similarity is lower than a preset similarity threshold, processing pixels of the mapping distribution region of the pixel block corresponding to each cosine similarity in the initial image according to a determination result and by using a binarization processing method to obtain a question visual relevant region map.

4. The image understanding method of claim 3, wherein, The method further comprises the following steps: marking pixels of the mapping distribution region of the pixel block corresponding to the cosine similarity not lower than the preset similarity threshold as 1; marking pixels of the mapping distribution region of the pixel block corresponding to the cosine similarity lower than the preset similarity threshold as 0.

5. The image understanding method of claim 1, wherein, The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes:

6. The image understanding method of claim 5, wherein, The perturbed image is obtained using the following equation: wherein, denotes the perturbation mask, denotes the image, denotes the perturbed image, denotes an element-wise multiplication operation.

7. The image understanding method of claim 1, wherein, The obtaining of the enhanced visual utilization rate of each enhanced image includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes:

8. An image understanding apparatus characterized by comprising: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area of the problem visual correlation area graph to the problem in the initial image for shielding or scrambling includes: The combining of the preset disturbance rule and the high correlation visual area The optimal visual expert selection module is used to call at least two types of visual experts to process the initial image when the visual utilization rate is lower than a preset utilization rate threshold, obtain the corresponding expert structured visual features, and use a visualization rendering function to superimpose each expert structured visual feature onto the initial image to obtain the corresponding enhanced image. Based on each enhanced image, the enhanced visual utilization rate of the corresponding visual expert is obtained. Based on the enhanced visual utilization rate of the visual expert and the visual utilization rate, the visual utilization rate improvement of each visual expert is calculated, and the visual expert with the largest visual utilization rate improvement is selected as the optimal visual expert. The reasoning result generation module is used to obtain the enhanced image corresponding to the best visual expert, generate an answer based on the question and the enhanced image corresponding to the best visual expert, and use the answer as the final image understanding reasoning result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image understanding method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by a processor, it implements the steps of the image understanding method as described in any one of claims 1 to 7.

Images