Image processing method and apparatus

By guiding a machine learning model to scan images region by region and generate observations, the visual shortcuts of multimodal models are suppressed, thereby improving the accuracy of image understanding and fine-grained perception capabilities.

CN123115447APending Publication Date: 2026-11-17ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611049741.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Multimodal models are prone to the visual shortcut problem, which is the generation of answers based on prior linguistic knowledge without relying on actual visual information.

Method used

By guiding the machine learning model to first generate observations of the image and then generating a response based on those observations, the system ensures that the response does not contain any uncovered visual information. Furthermore, the system provides prompts to scan the image region by region, perform self-checks, and preprocess to suppress visual shortcuts.

Benefits of technology

It effectively suppresses the visual shortcut problem of multimodal models, improves accuracy in fine-grained visual perception tasks, and ensures that the model generates responses based on the observation of images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115447A_ABST
    Figure CN123115447A_ABST
Patent Text Reader

Abstract

This specification provides one or more embodiments of an image processing method and apparatus. The method receives an image input by a user and a query regarding the image, then uses a machine learning model to perform image understanding processing to generate a response to the query. Specifically, during image understanding processing, visual perception processing can be performed on the image first to obtain observation results, and then a response to the query can be generated based on these observation results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer network technology, and in particular to an image processing method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence-related technologies, multimodal models are attracting increasing attention. Multimodal models enhance the versatility of models and their ability to understand the world more broadly by integrating and processing data from multiple modalities. They have wide applications in fields such as medical imaging, intelligent quality inspection, and artificial intelligence (AI) interviewing.

[0003] However, current multimodal models are prone to the visual shortcut problem, which involves generating answers based on prior linguistic knowledge without relying on actual visual information. Therefore, an image understanding method is needed to suppress visual shortcuts in multimodal models. Summary of the Invention

[0004] To suppress the visual shortcut problem present in multimodal models, one or more embodiments of this specification provide an image processing method and apparatus.

[0005] In a first aspect, one or more embodiments of this specification provide an image processing method, the method comprising: acquiring input data, the input data including an image input by a user and a query for the image, the query indicating a question related to the image; performing image understanding processing using a machine learning model based on the input data to generate a response to the query, wherein the image understanding processing includes: performing visual perception processing on the image to obtain an observation result of the image; and generating a response to the query based on the observation result, the response not containing visual information not covered by the observation result.

[0006] In one possible implementation, the input data further includes preset system prompts configured to guide the machine learning model to perform image understanding processing; the step of performing visual perception processing on the image to obtain the observation results of the image includes: using the system prompts to guide the machine learning model to scan the image region by region to obtain the observation results.

[0007] In one possible implementation, the observation results include at least one of the following: observation information of visible elements in the image, annotation information of abnormal regions in the image, and declaration information of uncertain elements in the image.

[0008] In one possible implementation, the observation information includes at least one of the following: the number, position, and pose information of visible elements in the image, the visibility information of each region in the image, the occlusion information of the image, and the quality information of the image.

[0009] In one possible implementation, the observation information, annotation information, declaration information, and response are presented in a preset order in the response.

[0010] In one possible implementation, the system prompt information is further configured to guide the machine learning model to perform a self-check for misleading factors in the image that affect the observation results.

[0011] In one possible implementation, the system prompt information is further configured to guide the machine learning model to re-observe the image from the perspective of first encountering the image when the observation results contain content contrary to existing knowledge.

[0012] In one possible implementation, the input data further includes preset system prompt information, and the method further includes: performing preprocessing on the image to add a marker to the image, wherein the system prompt information is further configured to instruct the machine learning model to reference the marker in the image in the response.

[0013] In one possible implementation, the method further includes: determining that the machine learning model did not generate the response based on observation of the image in response to the response not referencing the tag; and instructing the machine learning model to generate the response based on observation of the image in response to determining that the machine learning model did not generate the response based on observation of the image.

[0014] In one possible implementation, the observation result includes structured description information of the image, and the image understanding processing using a machine learning model includes a first execution stage and a second execution stage, which are independent of each other; wherein, the first execution stage includes: generating the structured description information based on the image using the machine learning model; the second execution stage includes: generating the response based on the structured description information and the query using the machine learning model.

[0015] In one possible implementation, the structured description information is in structured JSON format.

[0016] In one possible implementation, the method further includes: performing preprocessing on the image to introduce visual elements that contradict existing knowledge into the image, resulting in a counterintuitive image; and using the system prompt information to guide the machine learning model to generate the response based on the counterintuitive image.

[0017] In one possible implementation, the method further includes: obtaining a test set for the machine learning model, the test set including test images and queries against the test images; performing a test on the machine learning model using the test set to obtain test results; and determining a visual dependency metric value for the machine learning model based on the test results, wherein the visual dependency metric includes at least one of the following: description coverage, which indicates the ratio of the number of elements mentioned in the output of the machine learning model to the number of elements present in the input image; label hit rate, which indicates the proportion of times the output of the machine learning model correctly references labels in the input image, the labels being added to the test image through preprocessing; and anti-prior accuracy, which indicates the accuracy rate at which the machine learning model responds correctly when tested on a test set of images that violate common sense.

[0018] In one possible implementation, the method further includes: receiving user feedback on the response, the feedback instructing the user to select the correct response generated based on the observation results and reject incorrect responses generated based on prior knowledge; and guiding a machine learning model to generate the response based on the observation results based on the feedback.

[0019] In one possible implementation, the machine learning model includes a language model, which includes an N-layer decoder. The machine learning model is trained through the following process: obtaining the weights of visual lexical units in the nth-layer decoder of the language model, where n is greater than or equal to 2 and less than or equal to N, and N is a natural number greater than 2; and in response to the weights being lower than a preset weight threshold, instructing the language model to adjust the weights of the visual lexical units in the nth-layer decoder.

[0020] Secondly, one or more embodiments of this specification also provide an image processing apparatus, the apparatus comprising: an acquisition module configured to acquire input data, the input data including an image input by a user and a query for the image, the query indicating a question related to the image; and a processing module configured to perform image understanding processing based on the input data using a machine learning model to generate a response to the query, wherein the processing module is configured to: perform visual perception processing on the image to obtain an observation result of the image; and generate a response to the query based on the observation result, the response not containing visual information not covered by the observation result.

[0021] In one possible implementation, the input data further includes preset system prompts, which are configured to guide the machine learning model to perform image understanding processing; the processing module is configured to use the system prompts to guide the machine learning model to scan the image region by region to obtain the observation results.

[0022] In one possible implementation, the observation results include at least one of the following: observation information of visible elements in the image, annotation information of abnormal regions in the image, and declaration information of uncertain elements in the image.

[0023] In one possible implementation, the observation information includes at least one of the following: the number, position, and pose information of visible elements in the image, the visibility information of each region in the image, the occlusion information of the image, and the quality information of the image.

[0024] In one possible implementation, the observation information, annotation information, declaration information, and response are presented in a preset order in the response.

[0025] In one possible implementation, the system prompt information is further configured to guide the machine learning model to perform a self-check for misleading factors in the image that affect the observation results.

[0026] In one possible implementation, the system prompt information is further configured to guide the machine learning model to re-observe the image from the perspective of first encountering the image when the observation results contain content contrary to existing knowledge.

[0027] In one possible implementation, the input data further includes preset system prompt information, and the device further includes a preprocessing module configured to perform preprocessing on the image to add markers to the image, wherein the system prompt information is further configured to instruct the machine learning model to reference the markers in the image in the response.

[0028] In one possible implementation, the apparatus further includes a verification module configured to: determine that the machine learning model did not generate the response based on observation of the image in response to the response not referencing the tag; and instruct the machine learning model to generate the response based on observation of the image in response to determining that the machine learning model did not generate the response based on observation of the image.

[0029] In one possible implementation, the observation results include structured description information of the image, and the processing module is configured to: in a first execution phase, generate the structured description information based on the image using the machine learning model; in a second execution phase, generate the response based on the structured description information and the query using the machine learning model; wherein the first execution phase and the second execution phase are independent execution phases.

[0030] In one possible implementation, the structured description information is in structured JSON format.

[0031] In one possible implementation, the preprocessing module is further configured to: perform preprocessing on the image to introduce visual elements that contradict existing knowledge into the image, resulting in a counterintuitive image; and use the system prompt information to guide the machine learning model to generate the response based on the counterintuitive image.

[0032] In one possible implementation, the acquisition module is further configured to: acquire a test set for the machine learning model, the test set including test images and queries for the test images; the processing module is further configured to: perform a test on the machine learning model using the test set to obtain test results; and based on the test results, determine the value of a visual dependency metric of the machine learning model, wherein the visual dependency metric includes at least one of the following: description coverage, which indicates the ratio of the number of elements mentioned in the output of the machine learning model to the number of elements present in the input image; label hit rate, which indicates the proportion of times the output of the machine learning model correctly references labels in the input image, the labels being added to the test image by performing preprocessing on the test image; and anti-prior accuracy, which indicates the accuracy rate at which the machine learning model responds correctly when tested on a test set of images that violate common sense.

[0033] In one possible implementation, the device further includes a feedback module configured to: receive user feedback on the response, the feedback instructing the user to select the correct response generated based on the observation results and reject incorrect responses generated based on prior knowledge; and guide a machine learning model to generate the response based on the observation results based on the feedback.

[0034] In one possible implementation, the machine learning model includes an N-layer network, and the device further includes a training module configured to: obtain the weights of visual lexical units in the nth layer decoder of the language model, where n is greater than or equal to 2 and less than or equal to N, and N is a natural number greater than 2; and in response to the weights being lower than a preset weight threshold, instruct the language model to adjust the weights of the visual lexical units in the nth layer decoder.

[0035] Thirdly, one or more embodiments of this specification also provide an electronic device, which includes a memory and a processor; the memory is used to store a computer program product; the processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the image processing method of the first aspect described above.

[0036] Fourthly, one or more embodiments of this specification also provide a computer-readable storage medium storing computer program instructions that, when executed, implement the image processing method of the first aspect described above.

[0037] Fifthly, one or more embodiments of this specification also provide a computer program product, including computer-executable instructions, which, when executed by a processor, implement the image processing method of the first aspect described above.

[0038] In summary, to suppress the visual shortcut problem in multimodal models, one or more embodiments of this specification provide an image processing method and apparatus. In this method, after receiving an image input by a user and a query regarding the image, a machine learning model is used to generate a response to the query. When generating the response, visual perception processing is first performed on the image to obtain an observation result of the image. Then, a response to the query is generated based on the observation result, wherein the response does not contain visual information not covered by the observation result.

[0039] In this way, by guiding the machine learning model to first generate observations of the image and then generate responses to the query based on those observations, it is easier to encourage the model to truly generate responses to the query based on the observations of the image, thus suppressing the visual shortcut problem of the model. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of one or more embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of one or more embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram is shown in which an example environment in which one or more embodiments according to this specification may be implemented; Figure 2 A schematic diagram of an example architecture for image processing provided for one or more embodiments of this specification; Figure 3Another example architecture diagram for image processing provided for one or more embodiments of this specification; Figure 4 This is yet another example architecture diagram for image processing provided in one or more embodiments of this specification; Figure 5 A schematic diagram of an example architecture for a training system for a machine learning model provided in one or more embodiments of this specification; Figure 6 A schematic flowchart illustrating one or more embodiments of the image processing method provided in this specification; Figure 7 A structural block diagram of an image processing apparatus provided for one or more embodiments of this specification; Figure 8 This is a structural block diagram of an electronic device provided for one or more embodiments of this specification. Detailed Implementation

[0042] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0043] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0044] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0045] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0046] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0047] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0048] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0049] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0050] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0051] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0052] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0053] Terminology Explanation: Visual Shortcut: The behavior of a model generating an answer based on prior linguistic knowledge without relying on actual visual information.

[0054] A mirage is a phenomenon where a model can "confidently" describe the details of an image even without actually seeing it.

[0055] Semantic Prior Bias: The "concept-answer" mapping formed during the pre-training phase of the model causes it to rely on existing knowledge rather than on observation of the image. For example, when seeing a prompt related to "a cat sitting on a chair," the model can provide a reasonable description without actually understanding the image.

[0056] Visual attention inertia: In deep networks, visual tokens receive very low attention weights. For example, in some models, the average attention score of visual tokens after the second layer accounts for only 0.21% of the system prompts. Deeper layers of the model mainly rely on text tokens rather than visual information.

[0057] To facilitate understanding, the application scenarios of the technical solutions provided in one or more embodiments of this specification will be described below.

[0058] Figure 1 A schematic diagram of an example environment 100 in which one or an embodiment of this specification can be implemented is shown. In this example environment 100, an application 120 is installed on a terminal device 110. A user 140 can interact with the application 120 via the terminal device 110 and / or an attached device to the terminal device 110.

[0059] In some embodiments of this disclosure, application 120 can be any suitable application with human-computer dialogue capabilities. As an example, application 120 can provide a digital assistant for human-computer dialogue. This digital assistant supports content dialogue with user 140 based on text dialogue services, voice dialogue services, image dialogue services, video dialogue services, and other modalities.

[0060] In some embodiments of this disclosure, if application 120 is active, terminal device 110 may present user interface 150 of application 120. User interface 150 may include various pages that application 120 can provide, such as a dialogue page between the user and a digital assistant (where the current dialogue and historical dialogue, including text dialogue content, may be presented), and so on. In some embodiments, terminal device 110 may present text or images, and play voice or video in user interface 150.

[0061] In some embodiments, application 120 or its digital assistant may utilize machine learning model 160 (which may include one or more machine learning models, such as machine learning model 160-1, machine learning model 160-2, ..., machine learning model 160-N, etc., where N is a positive integer. For ease of description, the one or more machine learning models are collectively referred to as machine learning model 160 herein) to support interaction with user 140. For example, application 120 or its digital assistant may utilize one or more machine learning models 160 to understand images and generate response content for user 140 based on the understanding of the images.

[0062] Machine learning model 160 can be of different types. In some embodiments, one or more machine learning models 160 may be built based on a language model (LM). The machine learning model used is a content-generative model capable of generating corresponding outputs based on model inputs. In some embodiments, the language model-based machine learning model can understand multimodal model inputs (e.g., text, speech, images, videos, etc.) and can generate the desired output based on the model inputs and prompts. The prompts are used to guide the machine learning model to generate outputs that address the user needs indicated by the model inputs. In applications supporting user dialogue, user 140's input may be provided to machine learning model 160 as at least a part of the model inputs (other parts may include prompts). This user input is considered a query. Based on the model outputs, corresponding responses can be generated and provided to user 140. As an example, machine learning model 160 may be built based on a multimodal large language model (also referred to as a multimodal model) or a visual language model (VLM).

[0063] In some embodiments, terminal device 110 communicates with server 130 to provide services to application 120. For example... Figure 1 As shown, server 130 can invoke machine learning model 160 to support human-computer dialogue between application 120 and user 140 based on the output of machine learning model 160. Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of user-facing interface (such as "wearable" circuitry). Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc. Server 130 can, for example, be implemented in a cloud environment.

[0064] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0065] A serious problem in current multimodal models is that they can still "confidently" describe image details without actually seeing the image. Even when all images are quietly removed but the question remains, the model will still describe the image in detail in over 70% of cases. In some medical multimodal benchmarks, the model can even achieve high scores without looking at the image at all. This is mainly due to semantic prior bias and visual attention inertia. Furthermore, in tasks requiring fine-grained visual perception (such as palm / back of hand identification, gesture recognition, etc.), the visual encoder of current multimodal models cannot recognize key features (such as veins on the back of the hand, fingerprint patterns, and the main palm print), resulting in an accuracy of only about 16.5%, which is close to random guessing.

[0066] To suppress the visual shortcut problem in models and improve their accuracy on fine-grained visual perception tasks, this specification provides an image processing method and apparatus through one or more embodiments. This method can be applied to... Figure 2 The architecture shown is 200. Figure 3 The architecture 300 shown and Figure 4 The architecture shown is 400.

[0067] Figure 2This is a schematic diagram of an example architecture 200 for image processing provided in one or more embodiments of this specification.

[0068] like Figure 2 As shown, architecture 200 may include machine learning model 240. Machine learning model 240 is used to generate a response 250 to query 220 based on image 210, query 220, and prompt information 230.

[0069] In one possible implementation, the machine learning model 240 can be a visual language model (VLM), which exemplarily includes an image encoder, a visual projection component, and a language model. The image encoder is connected to the language model via the visual projection component. The image encoder is pre-trained and possesses general image encoding capabilities; it is used to encode the user-input image 210 to obtain the image encoding result. The visual projection component is used to perform feature transformation on the image encoding result to obtain a feature transformation result that can be understood by the language model; that is, the visual projection component is used to align the visual feature space of the image encoder to the text feature space of the language model. The language model is also pre-trained and possesses general language knowledge, world knowledge, and domain-specific expertise; it is used to process the feature transformation result according to the prompt information 230 to obtain a response 250 corresponding to the query 220 corresponding to the image 210. In other words, the language model generates the response 250 based on the visual lexical units corresponding to the input image 210 and the text lexical units corresponding to the query 220 and the prompt information 230.

[0070] In one possible implementation, query 220 indicates a question related to image 210. As an example, image 210 could be an image about animals, and query 220 could include questions such as how many animals are in the image.

[0071] In one possible implementation, the prompt message 230 is configured to guide the machine learning model 240 to generate a response 250 to the query 220 according to preset steps. The response 250 includes an answer to a question related to the image 210. For example, the prompt message 230 is configured to constrain the machine learning model 240 to first perform perceptual understanding of the image 210, and then generate the response 250 based on that perceptual understanding. The prompt message 230 aims to constrain the machine learning model 240 so that perception must precede all conclusions.

[0072] In one possible implementation, the prompt message 230 is configured for the machine learning model 240 to generate a response according to the following steps: Step 1 [Perception must precede all conclusions]: Scan image 210 region by region to obtain observation information of image 210. For example, scan image 210 region by region from the upper left corner to the lower right corner and generate observation information for image 210. Observation information may include the number, location, and pose information of visible elements / objects in image 210. Observation information may also include visibility information for each region in image 210; for example, the visibility of each region in image 210 may be listed individually in the observation information. Observation information may also include occlusion information of image 210, such as occluding objects in image 210 and their occlusion or coverage locations. Observation information may also include quality information of image 210, such as image 210 sharpness, brightness, and any anomalies.

[0073] Step 2 [Anomaly Labeling]: Generate annotation information for abnormal regions in image 210, that is, label abnormal regions in image 210, pointing out any regions in image 210 that are inconsistent with common sense, blurry, or may affect the judgment.

[0074] Step 3 [Uncertainty Declaration] generates declaration information for uncertain elements in image 210, that is, declares whether uncertain elements exist in image 210. If any element in image 210 cannot be confirmed, language model 218 must explicitly write: "I cannot confirm [specific element], because [specific reason]". If all elements in image 210 can be confirmed, language model 218 writes "No uncertain items".

[0075] Step 4 [Response]: Generate response 230 based on the observations from steps 1 to 3, prohibiting the introduction of information not observed in image 210.

[0076] In this way, the prompt information 230 guides the machine learning model 240 to generate a response 250 according to steps or rules. If the steps are violated or the rules are skipped, the response 250 is considered an invalid response. Furthermore, the above constraints can reduce the likelihood of the machine learning model 240 answering directly based on prior knowledge, thereby improving image understanding capabilities. In addition, since the machine learning model 240 is required to scan the image region by region, the accuracy of the model in fine-grained visual perception tasks can be improved.

[0077] In one possible implementation, the response 250 can be used to determine whether the machine learning model 240 generated the response 250 according to the steps or rules. For example, according to the constraints of the aforementioned prompt information 230, the observation information, annotation information, declaration information, and response in the response 250 should be presented in sequence. If the response 250 first describes that there are no uncertain items in image 210, and then describes that an element or object was observed in image 210, it can be determined that the language model 218 did not generate the response 250 according to the steps or rules. This is because, according to the constraints of prompt information 230, the element observed in image 210 should be described first in the response 250, followed by the annotation information and declaration information. As another example, if there is no annotation information for an abnormal region in the response 250, but the declaration information describes that it is impossible to confirm what a certain element in image 210 is because the region is blurry, it can be determined that the machine learning model 240 did not generate the response 250 according to the steps or rules.

[0078] In one possible implementation, the prompt message 230 is further configured to guide the machine learning model 240 to perform a self-check for misleading factors in the image 210 that affect the observation results. Optionally, the prompt message 230 can list traps that are likely to cause the machine learning model 240 to misjudge, and guide the model to perform a self-check for possible traps. Optionally, misleading factors can be, for example, color differences under different lighting intensities, such as black appearing brownish due to lighting. For such traps, the model can be prompted to consider environmental factors when seeing brown instead of directly identifying it as brown.

[0079] In one possible implementation, the prompt information 230 is further configured to guide the machine learning model 240 to re-observe image 210 from the perspective of its first encounter when the observation of image 210 contains content contrary to existing knowledge. For example, the price of a commodity in image 210 differs significantly from its usual price; for instance, the price of bottled water in image 210 is 200 yuan. Another example is that the horizontal axis in image 210 is labeled as the y-axis, and the vertical axis as the x-axis. In these cases, the prompt information 230 guides the machine learning model 240 to re-observe image 210 from the perspective of its first encounter.

[0080] Figure 3 This is a schematic diagram of an example architecture 300 for image processing provided in one or more embodiments of this specification.

[0081] like Figure 3As shown, architecture 300 may include an image preprocessing module 340 and a machine learning model 350. The image preprocessing module 340 performs preprocessing on image 310 to add markers to image 310. The machine learning model 350 generates a response 350 for query 320 based on the preprocessed image 310 and prompt information 330. Prompt information 330 is configured to guide the machine learning model 350 to reference all markers in image 310 in response 360. The image preprocessing module 340 can be various image editing tools used to add markers to image 310.

[0082] In one possible implementation, the machine learning model 350 can be a visual language model (VLM) or a language-based multimodal model.

[0083] In one possible implementation, the image preprocessing module 340 can be used to add markers at non-salient locations in the image 310. For example, markers can be added to corner areas or areas where the main object is not located in the image 310.

[0084] In one possible implementation, the tag can be a random string, such as a random combination of 8 alphanumeric characters, like ZYPHQ2026. In this scenario, the prompt message 330 can be configured to guide the machine learning model 350 to reference all combinations of 7 or more letters in the image 310 in the response 360. It should be understood that the random string is only one example of a tag, and other suitable tags can be added in practice depending on the application scenario or image type.

[0085] By adding labels to the image and guiding the multimodal model to reference these labels in its response, the presence or absence of labels in the response can determine whether the multimodal model generated the response 230 based on observations of the image. If no labels are present in the response 230, it can be determined that the machine learning model 350 did not generate the response 360 ​​based on observations of the image 310. That is, through... Figure 3 The proposed solution can determine whether the model is actually reading the image and respond accordingly, supporting fully automated verification of whether the model is truly reading or not.

[0086] In one possible implementation, the image preprocessing module 340 can also be used to preprocess the image 310 to introduce visual elements that contradict existing knowledge, resulting in a counterintuitive image. For example, visual elements that contradict existing knowledge may include swapped traffic light colors, interchanged horizontal and vertical axis labels on a chart, unusual price values ​​for common products, and fine-grained discrepancy traps (such as confusion between the characters "1" and "I", or RGB color fine-tuning). In this embodiment, the machine learning model 350 can be guided by prompt information 330 to generate a response based on the counterintuitive image.

[0087] By evaluating the observations or responses to images that violate common sense, it is possible to determine whether the machine learning model 350 relies excessively on prior knowledge rather than visual information.

[0088] Furthermore, images that violate common sense can be used to construct a test set to evaluate the visual dependence of machine learning model 350. For example, images that violate common sense can be input into machine learning model 350 as test images, and the accuracy of the machine learning model 350's output response can be used to determine its inverse prior accuracy metric. This inverse prior accuracy can be used as one of the visual dependence metrics to quantitatively evaluate the visual understanding ability of the machine learning model.

[0089] In one possible implementation, in response to the machine learning model 350 failing to generate a response 360 ​​based on the observation of image 310, the machine learning model 350 can be guided or instructed to regenerate the response 360 ​​based on the observation of image 310. For example, a prompt message 330 can be used to instruct the machine learning model 350 to regenerate the response 360 ​​based on the observation of image 310.

[0090] Figure 4 This is a schematic diagram of an example architecture 400 for image processing provided in one or more embodiments of this specification.

[0091] like Figure 4 As shown, in architecture 400, image understanding processing can include a first execution stage and a second execution stage. In the first execution stage, machine learning model 420 is invoked to process the user-input image 410, obtaining structured description information 430 of image 410. In the second execution stage, machine learning model 420 is invoked again to generate a response 450 for query 440 based on the structured description information 430 of image 410. That is, in this embodiment, the two-stage invocation of machine learning model 420 decouples visual perception and language reasoning, preventing the reasoning stage from bypassing the perception result and reducing the model's visual attention inertia. Furthermore, because visual perception and language reasoning are decoupled, errors in the perception stage can be audited independently.

[0092] In one possible implementation, the machine learning model 420 can be a visual language model (VLM) or a language-based multimodal model.

[0093] To mitigate the visual shortcut problem in models, one or more embodiments of this specification provide a method for training machine learning models. This method can be applied to... Figure 5 The architecture shown is 500.

[0094] Figure 5This is a schematic diagram of an example architecture 500 for a training system for a machine learning model provided in one or more embodiments of this specification.

[0095] like Figure 5 As shown, architecture 500 includes a machine learning model 530 and an attention monitoring module 540. Machine learning model 530 generates output from image 510 and input text 520. Input text 520 may include queries, prompts, and context or previously generated output for image 510. Output may include a response to the query. Machine learning model 530 may include multi-layered stacked backbone networks 530-1 to 530-N, which process visual lexical units formed based on input image 510 and textual lexical units formed based on input text to obtain output. During processing, the backbone network calculates attention weights for visual lexical units and textual lexical units; a higher attention weight for a visual lexical unit indicates a higher dependence of the layer's output on the visual lexical unit.

[0096] In one possible implementation, each backbone network layer can be considered a decoding layer, and each decoder layer may include an attention module and a feedforward neural network. The attention monitoring module 540 is used to monitor the attention weights of visual words in each backbone network layer. When the attention weight of a visual word falls below a set threshold, it instructs the reallocation of the attention weights of the visual word.

[0097] In one possible implementation, the attention monitoring module 540 is used to monitor the attention weights of visual words in the backbone network from the 3rd layer to the Nth layer, and when the attention weight of a visual word is lower than a set threshold, it instructs the backbone network weights of the visual word to be reallocated.

[0098] In one possible implementation, different visual word attention weight thresholds are assigned to each layer of the backbone network. For example, the earlier the backbone layer is, the larger the attention weight threshold for its visual words.

[0099] In one possible implementation, the attention weight thresholds for visual lexical units in the decoders from layer 3 to layer N are the same.

[0100] By monitoring the attention weights of visual lexical units at deeper levels of the multimodal model during training, the attention weights of visual lexical units after the second layer are kept above the threshold, allowing the model to still rely on visual lexical units to perform inference in deeper networks, which helps to suppress visual attention inertia in the model.

[0101] The following describes embodiments of the image processing method provided in one or more embodiments of this specification.

[0102] See Figure 6 , Figure 6This is a schematic flowchart illustrating an image processing method 600 provided in one or more embodiments of this specification. For ease of discussion, reference will be made to... Figure 1 The method 600 is described using the following environment: In environment 100, method 300 can be implemented at terminal device 110 and / or server 130. For ease of description, it is illustrated by the example of method 300 being implemented at server 130.

[0103] It should be noted that method 600 can be implemented at terminal device 110. Some operations described with reference to terminal device 110 may require the assistance of server 130. It should be noted that the operations performed by terminal device 110 may specifically be performed by relevant applications and / or digital assistants installed on terminal device 110.

[0104] like Figure 6 As shown, method 600 may include the following steps: Step 602: Server 130 obtains input data, which includes an image input by the user and a query for the image, wherein the query indicates a question related to the image.

[0105] by Figure 1 Taking environment 100 as an example, server 130 can obtain images input by user 140 to application 120 from terminal device 110, as well as queries about the images, which indicate questions related to the images.

[0106] In one possible implementation, the input data may further include preset system prompts, which can be configured to guide the machine learning model to perform image understanding processing. For details, please refer to the foregoing combination. Figure 2 , Figure 3 The description of the prompt message will not be repeated here.

[0107] Step 604: Server 130 performs image understanding processing based on the input data using machine learning model 160 to generate a response to the query. The image understanding processing includes: performing visual perception processing on the image to obtain the observation results of the image; and generating a response to the query based on the observation results. The response does not contain visual information not covered by the observation results, that is, when generating the response, it is prohibited to introduce visual information not covered by the observation results.

[0108] One possible implementation involves performing visual perception processing on the image to obtain the observation results, which may include: using system prompts to guide a machine learning model to scan the image region by region to obtain the observation results.

[0109] In one possible implementation, the machine learning model can be a visual language model or other multimodal models.

[0110] In one possible implementation, the response may include observations of the image and a response to the question. Optionally, the observations may include at least one of the following: observation information of visible elements in the image, annotation information of anomalous regions in the image, and declaration information of uncertain elements in the image. Optionally, the observation information may include at least one of the following: the number, position, and pose information of visible elements in the image, visibility information of each region in the image, occlusion information of the image, and quality information of the image. For details, please refer to the foregoing combination. Figure 2 The description will not be repeated here.

[0111] In one possible implementation, the observation information, annotation information, declaration information, and response are presented in a preset order in the response. This allows the system to determine whether the machine learning model generated the response according to preset steps or rules by observing the presentation order of the information in the response.

[0112] In one possible implementation, the system prompt information is further configured to guide the machine learning model to perform a self-check for misleading factors in the image that affect the observation results. For details, please refer to the foregoing combination. Figure 2 The description will not be repeated here.

[0113] In one possible implementation, the system prompt information is further configured to guide the machine learning model to re-observe the image from the perspective of its first encounter when the observation contains content contrary to existing knowledge. For details, please refer to the foregoing combination. Figure 2 The description will not be repeated here.

[0114] In one possible implementation, method 600 further includes: performing preprocessing on the image to add tags to the image, wherein the system prompt information is further configured to instruct the machine learning model to reference the tags in the image in the response. See the foregoing for details. Figure 3 The description will not be repeated here.

[0115] In one possible implementation, method 600 further includes: determining that the machine learning model did not generate the response based on observation of the image in response to the response not referencing the tag; and instructing the machine learning model to generate the response based on observation of the image in response to determining that the machine learning model did not generate the response based on observation of the image. For details, please refer to the foregoing combination. Figure 3 The description will not be repeated here.

[0116] In one possible implementation, the observation result includes structured description information of the image, and the image understanding processing using a machine learning model may include a first execution stage and a second execution stage, which are independent of each other. The first execution stage includes generating the structured description information based on the image using a machine learning model; and the second execution stage includes generating the response based on the structured description information and the query using a machine learning model. For details, please refer to the foregoing combination. Figure 4 The description will not be repeated here.

[0117] In one possible implementation, the structured description information is in structured JSON format.

[0118] In one possible implementation, method 600 further includes: performing preprocessing on the image to introduce visual elements contrary to existing knowledge, resulting in a counterintuitive image; and using the system prompt information to guide the machine learning model to generate the response based on the counterintuitive image. For details, please refer to the foregoing combination. Figure 3 The description will not be repeated here.

[0119] In one possible implementation, method 600 further includes: acquiring a test set for the machine learning model, the test set including test images and queries against the test images; performing a test on the machine learning model using the test set to obtain test results; and determining, based on the test results, a value for a visual dependency metric of the machine learning model, wherein the visual dependency metric includes at least one of the following: description coverage, which indicates the ratio of the number of elements mentioned in the output of the machine learning model to the number of elements present in the input test image; label hit rate, which indicates the proportion of times the output of the machine learning model correctly references labels in the input test image, the labels being added to the test image through preprocessing; and anti-prior accuracy, which indicates the accuracy rate at which the machine learning model correctly responds when tested on a test set of images that violate common sense. This measures the model's visual understanding ability from different dimensions using three complementary metrics, and these three metrics can directly serve as preference signals for model training / fine-tuning to improve the model's visual understanding ability and suppress the model's visual shortcut problem.

[0120] In one possible implementation, method 600 further includes: receiving feedback from a user on the response, the feedback instructing the user to select the correct response generated based on the observation and to reject the incorrect response generated based on prior information; and guiding a machine learning model based on the feedback to generate the response based on the observation.

[0121] In this way, selecting the correct response generated based on the observations through user feedback and rejecting incorrect responses generated based on prior knowledge can guide the model to answer user questions based on the observation of the image, rather than directly relying on prior knowledge to answer user questions.

[0122] In one possible implementation, the machine learning model may include an N-layer network, which is trained through the following process: obtaining the attention weights of visual words in the nth layer of the machine learning model, where n is greater than or equal to 2 and less than or equal to N, and N is a natural number greater than 2; and in response to the attention weights falling below a preset weight threshold, instructing the adjustment of the attention weights of the visual words in the nth layer. For details, please refer to the foregoing combination. Figure 5 The description will not be repeated here.

[0123] In one possible implementation, the image processing method provided in one or more embodiments of this specification can be applied to AI interview scenarios, such as enhancing image understanding in application scenarios like candidate identity verification, gesture detection, and liveness detection. In this scenario, the input image can be a candidate's face image, gesture image, etc. The image processing method provided in one or more embodiments of this specification can also be applied to intelligent quality inspection scenarios, such as image detection and defect identification in industrial production that require high-precision image understanding. In this scenario, the input image can be various workpieces or products in industrial production. The image processing method provided in one or more embodiments of this specification can also be applied to medical image analysis scenarios, such as assisting in medical image diagnosis to ensure that the model truly reads the image rather than relying on prior clinical knowledge. In this scenario, the input image can be medical examination images. The image processing method provided in one or more embodiments of this specification can also be applied to intelligent security monitoring scenarios, such as face recognition and behavior analysis that require accurate visual perception. In this scenario, the input image can be a face image, a human motion image, or a video. The image processing method provided in one or more embodiments of this specification can also be applied to autonomous driving perception scenarios to improve the accurate understanding of the road environment by the onboard vision system. In this scenario, the input image can be a road scene image collected by the vehicle. The image processing method provided in one or more embodiments of this specification can also be applied to content moderation scenarios, such as security moderation of image content, to prevent model misjudgment. In this scenario, the input image can be various images that need to be moderated, such as images or videos uploaded to online platforms.

[0124] In the image processing method provided in one or more embodiments of this specification, the machine learning model is guided to first scan the image region by region to generate observation results for the image, and then generate a response to the query based on the observation results. This facilitates the model to truly generate a response to the query based on the observation of the image, thus suppressing the visual shortcut problem of the model. Specifically, the image processing method provided in one or more embodiments of this specification uses a four-step structured visual perception process to encourage the model to perform spatial and systematic image observation, and introduces an "uncertainty declaration" mechanism to make the model acknowledge visual blind spots, which helps to suppress the visual shortcut problem of the model.

[0125] Furthermore, the image processing method provided in one or more embodiments of this specification introduces markers into the input image, which can automatically verify whether the model is truly visual. This can verify whether the model uses visual information without manual annotation and can be used to continuously monitor model behavior in the production environment.

[0126] Furthermore, the image processing method provided in one or more embodiments of this specification can adopt a two-stage physical isolation architecture, completely separating visual perception and language reasoning into independent calls, achieving the purpose of error auditability and process traceability, and can be applied to any multimodal model without retraining.

[0127] Furthermore, the image processing method provided in one or more embodiments of this specification proposes a Visual Anchoring Score (VGS) evaluation system, which presents a systematic quantitative evaluation framework for visual understanding. Three complementary indicators measure visual understanding ability from different dimensions and can be directly used as preference signals for model training / fine-tuning.

[0128] It is understood that the above embodiments are merely examples, and modifications can be made to the above embodiments in actual implementation. Those skilled in the art will understand that any modifications to the above embodiments that do not require creative effort fall within the protection scope of one or more embodiments of this specification, and will not be described again in the embodiments.

[0129] Based on the same inventive concept, one or more embodiments of this specification also provide an image processing apparatus. Since the principle of the problem solved by the image processing apparatus is similar to that of the aforementioned image processing method, the implementation of the image processing apparatus can refer to the implementation of the aforementioned image processing method, and repeated details will not be described again.

[0130] See Figure 7 , Figure 7 This is a structural block diagram of an image processing apparatus provided for one or more embodiments of this specification. Figure 7As shown, the image processing apparatus 700 may include an acquisition module 710 and a processing module 720. The acquisition module 710 is configured to acquire input data, including an image input by a user and a query regarding the image, the query indicating a question related to the image. The processing module 720 is configured to perform image understanding processing using a machine learning model based on the input data, generating a response to the query, wherein the processing module 720 is used to: perform visual perception processing on the image to obtain an observation result of the image; and generate a response to the query based on the observation result, the response not containing visual information not covered by the observation result.

[0131] In one possible implementation, the input data further includes preset system prompts, which are configured to guide the machine learning model to perform image understanding processing; the processing module is configured to use the system prompts to guide the machine learning model to scan the image region by region to obtain the observation results.

[0132] In one possible implementation, the observation results include at least one of the following: observation information of visible elements in the image, annotation information of abnormal regions in the image, and declaration information of uncertain elements in the image.

[0133] In one possible implementation, the observation information includes at least one of the following: the number, position, and pose information of visible elements in the image, the visibility information of each region in the image, the occlusion information of the image, and the quality information of the image.

[0134] In one possible implementation, the observation information, annotation information, declaration information, and response are presented in a preset order in the response.

[0135] In one possible implementation, the system prompt information is further configured to guide the machine learning model to perform a self-check for misleading factors in the image that affect the observation results.

[0136] In one possible implementation, the system prompt information is further configured to guide the machine learning model to re-observe the image from the perspective of first encountering the image when the observation results contain content contrary to existing knowledge.

[0137] In one possible implementation, the apparatus 700 further includes an image preprocessing module configured to perform preprocessing on the image to add tags to the image, wherein the system prompt information is further configured to instruct the machine learning model to reference the tags in the image in the response.

[0138] In one possible implementation, the apparatus 700 further includes a verification module configured to: determine that the machine learning model did not generate the response based on observation of the image in response to the response not referencing the tag; and instruct the machine learning model to generate the response based on observation of the image in response to determining that the machine learning model did not generate the response based on observation of the image.

[0139] In one possible implementation, the observation results include structured description information of the image, and the processing module 720 is configured to: in a first execution phase, generate the structured description information based on the image using a machine learning model; and in a second execution phase, generate the response based on the structured description information and the query using a machine learning model; wherein the first execution phase and the second execution phase are independent execution phases.

[0140] In one possible implementation, the structured description information is in structured JSON format.

[0141] In one possible implementation, the processing module 720 is further configured to: perform preprocessing on the image to introduce visual elements that contradict existing knowledge into the image, resulting in a counterintuitive image; and use the system prompt information to guide the machine learning model to generate the response based on the counterintuitive image.

[0142] In one possible implementation, the acquisition module 710 is further configured to acquire a test set for the machine learning model, the test set including test images and queries against the test images. The processing module 720 is further configured to perform a test on the machine learning model using the test set to obtain test results; and based on the test results, determine a visual dependency metric value for the machine learning model, wherein the visual dependency metric includes at least one of the following: description coverage, indicating the ratio of the number of elements mentioned in the output of the machine learning model to the number of elements present in the input test image; label hit rate, indicating the proportion of times the output of the machine learning model correctly references labels in the input test image, the labels being added to the test image through preprocessing; and anti-prior accuracy, indicating the accuracy rate at which the machine learning model correctly responds when tested on a non-common sense image test set.

[0143] In one possible implementation, the device 700 further includes a feedback module configured to receive feedback from a user on the response, the feedback instructing the user to select the correct response generated based on the observation and to reject the incorrect response generated based on prior knowledge; and to guide a machine learning model based on the feedback to generate the response based on the observation.

[0144] In one possible implementation, the machine learning model includes an N-layer network, and the device 700 further includes a training module configured to: obtain the attention weights of visual words in the nth layer of the machine learning model, where n is greater than or equal to 2 and less than or equal to N, and N is a natural number greater than 2; and, in response to the attention weights being lower than a preset weight threshold, instruct the adjustment of the attention weights of the visual words in the nth layer of the network.

[0145] See Figure 8 , Figure 8 This is a structural block diagram of an electronic device provided for one or more embodiments of this specification. Figure 8 As shown, the electronic device 800 may include a processor 810 and a memory 820; the memory 820 may be coupled to the processor 810. It is worth noting that... Figure 8 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0146] In one possible implementation, the functionality of the image processing device 700 can be integrated into the processor 810. The processor 810 can be configured to perform the following operations: acquiring input data, including a user-input image and a query for the image, the query indicating a question related to the image; and based on the input data, performing image understanding processing using a machine learning model to generate a response to the query, wherein the image understanding processing includes: performing visual perception processing on the image to obtain an observation result of the image; and generating a response to the query based on the observation result, the response not containing visual information not covered by the observation result.

[0147] In another possible implementation, the image processing device 700 can be configured separately from the processor 810. For example, the image processing device 700 can be configured as a chip connected to the processor 810, and image processing can be implemented under the control of the processor 810.

[0148] Furthermore, in some alternative implementations, the electronic device 800 may also include: a communication module, an input unit, an audio processor, a display, a power supply, etc. It is worth noting that the electronic device 800 is not necessarily required to include these components. Figure 8 All components shown; in addition, the electronic device 800 may also include Figure 8 For components not shown, please refer to existing technologies.

[0149] In some alternative implementations, the processor 810, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 800.

[0150] The memory 820 may be, for example, one or more of a cache, flash memory, hard drive, removable medium, volatile memory, non-volatile memory, or other suitable device. It can store the aforementioned information related to the image processing apparatus 700, and may also store programs for executing that information. The processor 810 can execute the program stored in the memory 820 to perform information storage or processing, etc.

[0151] An input unit can provide input to the processor 810. This input unit may be, for example, a button or touch input device. A power supply can be used to provide power to the electronic device 800. A display can be used to display images and text, etc. This display may be, for example, an LCD display, but is not limited to this.

[0152] The memory 820 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs, etc. The memory 820 can also be some other type of device. The memory 820 includes a buffer memory (sometimes called a buffer). The memory 820 may include an application / function storage unit for storing application programs and function programs or processes for executing the operation of the electronic device 800 via the processor 810.

[0153] The memory 820 may also include a data storage section for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage section of the memory 820 may include various drivers for the computer device for communication functions and / or for performing other functions of the computer device (such as messaging applications, address book applications, etc.).

[0154] The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor 810 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0155] Based on different communication technologies, multiple communication modules can be configured in the same computer device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication modules (transmitters / receivers) are also coupled to speakers and microphones via an audio processor to provide audio output through the speakers and receive audio input from the microphones, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor is coupled to processor 810, enabling on-device recording via the microphone and on-device playback of stored sound via the speakers.

[0156] One or more embodiments of this specification also provide a computer-readable storage medium capable of implementing all steps of the image processing methods in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the image processing methods in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: acquiring input data, the input data including an image input by a user and a query for the image, the query indicating a question related to the image; and performing image understanding processing based on the input data using a machine learning model to generate a response to the query, wherein the image understanding processing includes: performing visual perception processing on the image to obtain an observation result of the image; and generating a response to the query based on the observation result, the response not containing visual information not covered by the observation result.

[0157] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual device or client product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0158] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, apparatus (systems), or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This specification describes one or more embodiments of a method, apparatus (system), and computer program product according to one or more embodiments of this specification with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and system embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0163] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Those skilled in the art will understand the specific meaning of the above terms in one or more embodiments of this specification, depending on the specific circumstances.

[0164] It should be noted that, unless otherwise specified, one or more embodiments and features thereof in this specification can be combined with each other. This specification is not limited to any single aspect, nor to any single embodiment, nor to any combination and / or substitution of such aspects and / or embodiments. Furthermore, each aspect and / or embodiment of one or more embodiments of this specification can be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0165] It should also be noted that the relevant user personal information that may be involved in one or more embodiments of this specification is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the application scenario, and is personal information that users actively provide or that is generated due to the use of the product / service, as well as personal information obtained with the user's authorization.

[0166] The user personal information processed in one or more embodiments of this specification may vary depending on the specific product / service scenario and should be determined based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat user personal information and its processing with the utmost diligence.

[0167] This specification and one or more embodiments place great emphasis on the security of users' personal information and have adopted reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of one or more embodiments of this specification, and are not intended to limit them. Although one or more embodiments of this specification have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of one or more embodiments of this specification, and they should all be covered within the scope of the claims and the specification of one or more embodiments of this specification.

[0169] The foregoing description of one or more embodiments of this specification has been provided in conjunction with optional implementation methods. However, these embodiments are merely exemplary and serve only an illustrative purpose. Based on this, various substitutions and modifications can be made to one or more embodiments of this specification, all of which fall within the protection scope of one or more embodiments of this specification.

Claims

1. An image processing method, characterized in that, The method includes: Obtain input data, which includes an image input by the user and a query for the image, the query indicating a question related to the image; Based on the input data, an image understanding process is performed using a machine learning model to generate a response to the query. The image understanding processing mentioned above includes: Visual perception processing is performed on the image to obtain the observation results of the image; A response to the query is generated based on the observations, and the response does not include visual information not covered by the observations.

2. The method as described in claim 1, characterized in that, The input data also includes preset system prompts, which are configured to guide the machine learning model to perform image understanding processing; the visual perception processing of the image to obtain the observation results of the image includes: Using the system prompts, the machine learning model is guided to scan the image region by region to obtain the observation results.

3. The method as described in claim 1, characterized in that, The observations include at least one of the following: Observational information of visible elements in the image. The annotation information of abnormal regions in the image The declaration information of uncertain elements in the image.

4. The method as described in claim 3, characterized in that, The observation information includes at least one of the following: The number, position, and pose information of visible elements in the image. The visibility information of each region in the image The occlusion information of the image, The image quality information.

5. The method as described in claim 3, characterized in that, The response presents the observation information, annotation information, declaration information, and response in a preset order.

6. The method as described in claim 2, characterized in that, The system prompts are also configured to guide the machine learning model to perform a self-check for misleading factors in the image that affect the observation results.

7. The method as described in claim 2, characterized in that, The system prompt is also configured to guide the machine learning model to re-observe the image from the perspective of first encountering the image when the observation results contain content contrary to existing knowledge.

8. The method as described in claim 1, characterized in that, The input data also includes preset system prompts, and the method further includes: Preprocessing is performed on the image to add markers to the image. The system prompt information is configured to instruct the machine learning model to reference the markers in the image in the response.

9. The method as described in claim 8, characterized in that, The method further includes: If the response does not reference the tag, it is determined that the machine learning model did not generate the response based on observations of the image; In response to determining that the machine learning model did not generate the response based on observations of the image, the machine learning model is instructed to generate the response based on observations of the image.

10. The method as described in claim 1, characterized in that, The observation results include structured descriptive information of the image, and the image understanding processing using a machine learning model includes a first execution stage and a second execution stage, which are independent of each other; wherein... The first execution phase includes: generating the structured description information based on the image using the machine learning model; The second execution phase includes: generating the response using the machine learning model based on the structured description information and the query.

11. The method as described in claim 10, characterized in that, The structured description information is in structured JSON format.

12. The method as described in claim 2, characterized in that, The method further includes: The image is preprocessed to introduce visual elements that contradict existing knowledge, resulting in an image that defies common sense. Using the system prompts, the machine learning model is guided to generate the response based on the image that violates common sense.

13. The method as described in claim 1, characterized in that, The method further includes: Obtain a test set for the machine learning model, the test set including test images and queries against the test images; The machine learning model is tested using the test set, and the test results are obtained. Based on the test results, the index value of the visual dependency index of the machine learning model is determined. The visual dependency metrics mentioned above include at least one of the following: The descriptive coverage indicates the ratio of the number of elements mentioned in the output of the machine learning model to the number of elements present in the input test image; The label hit rate indicates the proportion of labels in the output of the machine learning model that correctly reference the input test image, the labels being added to the test image by performing preprocessing on the test image; Anti-prior accuracy, which indicates the accuracy with which the machine learning model responds correctly when tested on a set of images that violate common sense.

14. The method as described in claim 1, characterized in that, The method further includes: Receive user feedback on the response, the feedback instructing the user to select the correct response generated based on the observation, and to reject the incorrect response generated based on prior knowledge; The feedback guides the machine learning model to generate the response based on the observations.

15. The method as described in claim 1, characterized in that, The machine learning model comprises an N-layer network and is trained through the following process: Obtain the attention weights of visual words in the nth layer of the machine learning model, where n is greater than or equal to 2 and less than or equal to N, and N is a natural number greater than 2; In response to the attention weight being lower than a preset weight threshold, the attention weight of the visual word in the nth layer network is adjusted.

16. An image processing apparatus, characterized in that, The device includes: The acquisition module is configured to acquire input data, which includes an image input by a user and a query for the image, the query indicating a question related to the image; The processing module is configured to perform image understanding processing using a machine learning model based on the input data, and generate a response to the query. The processing module is used for: Visual perception processing is performed on the image to obtain the observation results of the image; A response to the query is generated based on the observations, and the response does not include visual information not covered by the observations.

17. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed, implement the method described in any one of claims 1 to 15.