A desktop chat multi-modal reply method based on picture placeholder positioning and secondary visual interpretation

Abstract

The application relates to the technical field of intelligent interaction, and provides a desktop chat multi-modal reply method based on picture placeholder positioning and secondary visual interpretation, which comprises the following steps: in response to reading a picture class attachment placeholder from a chat context and meeting a trigger condition, performing conservative region cutting on a chat screenshot to obtain a picture evidence cutting graph; calling a visual model to perform structured visual interpretation on the picture evidence cutting graph, cleaning an interpretation result, and generating a picture evidence with a confidence label; supplementing keywords in the picture evidence to a knowledge retrieval query to expand a retrieval recall range; and injecting the picture evidence as weak evidence into a reply generation prompt together with preset use limitation conditions. The application can significantly improve chat picture scene knowledge retrieval recall and reply quality, eliminate the risk of visual model output being mistaken for a certain fact, ensure that a basic reply process is not affected when multi-modal capability fails, realize the unity of intelligent enhancement and safety control, and the like.

Description

Technical Field

[0001] This invention relates to the field of intelligent interaction technology, and in particular to a multimodal reply method for desktop chat based on image placeholder positioning and secondary visual interpretation. Background Technology

[0002] With the widespread adoption of digital office solutions in enterprises, desktop instant messaging has become a core medium for daily work communication. To improve work efficiency, desktop chat assistants typically first perform screenshot recognition of the chat window, extract text messages, search the knowledge base, and generate suggested responses. However, users extensively transmit information via image messages in their work—error screenshots, table images, contract images, invoice images, or flowcharts, etc.—and existing technologies are clearly insufficient for handling such scenarios.

[0003] Existing text recognition technologies can only convert image messages into placeholder text such as "image" or "screenshot," completely failing to perceive the key information within the image. If an entire chat screenshot is directly fed into the visual model, irrelevant areas such as the top title, left-hand conversation list, and bottom input box will severely interfere with the model's understanding, causing it to mistakenly use irrelevant content as the basis for questions. Furthermore, chat screenshots often suffer from quality issues such as compression, obscuring, thumbnails, or partial display. If visual interpretations are directly used as definitive outputs, it can lead to the incorrect transmission of sensitive information such as amounts, times, numbers, or commitments, potentially causing serious consequences in financial, contractual, and compliance scenarios. Image content may also be inconsistent with knowledge base information; user-sent screenshots may be historical versions or partial captures. If the knowledge base is not used as a reference, responses will not conform to official company regulations. Additionally, the visual model itself is prone to instability such as timeouts or anomalies. If it becomes rigidly dependent on the basic response process, it will reduce system availability.

[0004] Chinese patent CN122414418A discloses an interaction method based on a multimodal large language model. This method intervenes in the training process of the multimodal large language model's preference alignment with data, forcing the model to rely on real visual evidence rather than linguistic priors for responses during training, thereby improving the accuracy and credibility of visual interaction tasks. However, this method only addresses the visual dependency issue from the perspective of data sampling and preference optimization during model training. It does not address the engineering control mechanisms for confidence grading, evidence health assessment, and failure degradation of the visual model's output results during the inference application stage. Furthermore, it does not propose solutions for specific application aspects such as image placement in desktop chat screenshot scenarios, conservative cropping of text areas, and weak evidence injection for response generation, thus lacking guarantees for the safe use of visual understanding results.

[0005] Therefore, there is an urgent need for a multimodal response method that can supplement visual understanding by starting with image placeholders while maintaining safety boundaries: it should be triggered only when there is a recently existing image placeholder and a screenshot is available, conservatively crop the text area to reduce interference, classify the confidence level of the image interpretation, participate in knowledge retrieval and response generation only as weak evidence, and only be used for follow-up questions when the confidence level is low. Furthermore, it should not affect the basic text response process when visual understanding fails, thereby achieving a unity of intelligent enhancement and risk control. Summary of the Invention

[0006] In view of this, in order to overcome the shortcomings of the prior art, the present invention aims to provide a desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation.

[0007] This invention provides a desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation, the method comprising:

[0008] Step S1: In response to reading an image attachment placeholder from the chat context and meeting the triggering conditions, perform conservative region cropping on the chat screenshot to obtain a cropped image evidence;

[0009] Step S2: Call the visual model to perform structured visual interpretation of the cropped image evidence, clean the interpretation results, and generate image evidence with confidence labels;

[0010] Step S3: Add keywords from the image evidence to the knowledge retrieval query to expand the retrieval scope, and inject the image evidence as weak evidence along with preset usage restrictions into the response to generate prompts.

[0011] Optionally, in the desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation of the present invention, in step S1, image attachment placeholders are read from the chat context in the following manner: non-text messages in the chat message are converted into placeholder text through chat screenshot recognition, and the placeholder type, placeholder tag, message number, speaker and placeholder description are recorded as attachment placeholder clues, wherein image placeholders include at least image placeholders and screenshot placeholders.

[0012] Optionally, in the desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation of the present invention, the triggering conditions in step S1 include: there are image placeholders in a preset number of messages within a preset time range, the current screenshot file is readable, and the visual model configuration is valid.

[0013] Optionally, in the desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation of the present invention, in step S1, conservative region cropping of the chat screenshot is performed in the following manner: the image width and height of the chat screenshot are obtained, the cropping frame is determined according to a preset boundary ratio, and cropping is performed after boundary correction of the cropping frame coordinates to obtain an image evidence cropping image that covers the lower area of ​​the chat text and avoids the top title, the left conversation list and the bottom input box.

[0014] Optionally, in the desktop chat multimodal response method based on image placeholder positioning and secondary visual interpretation of the present invention, in step S2, the visual model is invoked to perform structured visual interpretation of the image evidence cropping image in the following manner: the image evidence cropping image and image placeholder source clues are sent to the visual model, and the visual model outputs a structured interpretation result including image type, summary, key fact list, keyword list, confidence level, and whether user confirmation is required. The image type is selected from one of the following: error screenshot, table, contract, invoice, flowchart, ordinary image, and unknown type.

[0015] Optionally, in the desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation of the present invention, in step S2, the interpretation results are cleaned in the following manner: verifying whether the image type belongs to a preset type set, and if not, setting it to an unknown type; normalizing the confidence level to the range of 0 to 1; removing empty strings, deduplicating, and limiting the number of elements in the keyword list and the key fact list respectively; and replacing the summary with conservative summary content when the summary is empty.

[0016] Optionally, in the desktop chat multimodal response method based on image placeholder positioning and secondary visual interpretation of the present invention, step S2 generates image evidence with confidence level markers as follows: An image evidence object is constructed based on the cleaned interpretation results. This image evidence object includes at least image type, summary, confidence level, cropped image path, source clues, health level, health description, keywords, key facts, user confirmation marker, and action prompt. When the confidence level is lower than a preset low confidence threshold, the user confirmation marker is forced to indicate that confirmation is required, the health level is set to low health, and the health description limits the image evidence to only be used for follow-up confirmation. When the confidence level is not lower than the preset low confidence threshold, the user confirmation marker is set to indicate that confirmation is not required, the health level is set to ordinary weak evidence, and the health description indicates that the image evidence needs to be verified in conjunction with the chat context and knowledge base.

[0017] Optionally, in the desktop chat multimodal response method based on image placeholder positioning and secondary visual interpretation of the present invention, in step S3, keywords from the image evidence are added to the knowledge retrieval query to expand the retrieval recall scope in the following manner: the original user question is retained, and the keywords are appended to the original user question according to the deduplication rule to form an extended retrieval query; the keywords are only used to expand the retrieval scope and do not indicate that the image content has been strongly confirmed, and the image summary and key facts are not directly written into the knowledge base as confirmed facts.

[0018] Optionally, in step S3 of the desktop chat multimodal response method based on image placeholder positioning and secondary visual interpretation of the present invention, the preset usage restrictions include: image evidence is weak evidence; image evidence with a confidence level lower than a preset low confidence threshold can only be used for follow-up questions for confirmation; follow-up questions are given priority when image evidence conflicts with knowledge base retrieval results; generating any factual data, entity identifiers, or subjective conventions based on image evidence without structured interpretation support is prohibited; when the image type or keywords of the image evidence indicate the existence of structured documents or voucher-like information that need to be verified, an action prompt is generated to guide the user to verify the source of the external document.

[0019] Optionally, in the desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation of the present invention, when any step S1 to S3 fails, the use of image evidence is terminated, the original text recognition, knowledge retrieval and reply generation process is executed, and a prompt message for supplementing the original image or manual confirmation is output.

[0020] The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation of the present invention has the following beneficial technical effects:

[0021] Instead of treating image messages as empty placeholder text, it can extract structured information such as image type, summary, key facts and keywords, significantly improving the ability to respond in image message scenarios;

[0022] By using conservative region cropping, interference from irrelevant areas such as the top title, left-side conversation list, and bottom input box in chat screenshots is effectively eliminated, improving the purity of the input information for the visual model.

[0023] By consistently labeling image interpretations as weak evidence and imposing usage restrictions, we prevent visual model outputs from being mistaken for established facts, thus maintaining a safe boundary for response generation.

[0024] By supplementing image keywords into knowledge retrieval queries, the retrieval recall effect is improved when the original question information is insufficient, enabling even ambiguous questions to hit relevant knowledge. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to the present invention.

[0027] Figure 2 This is a flowchart illustrating the trigger judgment process for a desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to an exemplary embodiment of the present invention.

[0028] Figure 3 This is a flowchart illustrating the process of cleaning the interpretation results in a desktop chat multimodal response method based on image placeholder positioning and secondary visual interpretation according to an exemplary embodiment of the present invention.

[0029] Figure 4 This is a flowchart illustrating the process of generating image evidence with confidence markers using a desktop chat multimodal response method based on image placeholder positioning and secondary visual interpretation according to an exemplary embodiment of the present invention. Detailed Implementation

[0030] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0032] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0033] An exemplary embodiment of the present invention provides a multimodal reply method for desktop chat based on image placeholder positioning and secondary visual interpretation. Figure 1 The following is a flowchart illustrating the desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to the present invention. Figure 1 As shown, in this embodiment, the method of the present invention is implemented in the following manner:

[0034] Step S1: In response to reading an image attachment placeholder from the chat context and meeting the triggering conditions, perform conservative region cropping on the chat screenshot to obtain a cropped image evidence.

[0035] This embodiment records complete attachment placeholder clues, including placeholder type, placeholder tag, associated message number, speaker, and placeholder description, to provide accurate trigger source information for subsequent secondary understanding of images. As an optional example, this embodiment reads image attachment placeholders from the chat context in the following way: non-text messages in the chat message are converted into placeholder text through chat screenshot recognition, and the placeholder type, placeholder tag, associated message number, speaker, and placeholder description are recorded as attachment placeholder clues. Among them, image placeholders include at least image placeholders and screenshot placeholders.

[0036] This embodiment can accurately distinguish image placeholders from other non-text placeholders (such as emoticons, voice, files, etc.), avoiding invalid visual understanding calls for non-image placeholders; at the same time, the completeness of the source clues enables the visual model to know the position of the image in the chat context and the sender's identity, which helps the model to make more accurate visual interpretations in combination with the dialogue context, and also provides traceable evidence for subsequent evidence tracing and problem investigation.

[0037] In this embodiment, the visual model refers to a multimodal artificial intelligence model capable of receiving image input and performing semantic understanding and structured output of image content. Its core technical capability lies in converting visual signals (pixel matrices) into semantic information that can be parsed and utilized by computer systems, including the object types, text information, layout structure, semantic relationships, and other content contained in the image. Examples include general multimodal large models (GPT-4V, Qwen-VL), dedicated visual language models, and OCR-enhanced visual understanding models.

[0038] Figure 2 This is a flowchart illustrating the trigger judgment process for a desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to an exemplary embodiment of the present invention. Figure 2 As shown, in this embodiment, the triggering conditions include: there is an image placeholder in a preset number of messages within a preset time range, the current screenshot file is readable, and the visual model configuration is valid.

[0039] This embodiment focuses visual understanding resources on the latest image messages near the user's current question by setting filtering conditions of "preset time range and preset quantity", effectively avoiding accidental triggering and resource waste of historical images; by requiring that the screenshot file be readable and the visual model configuration be valid, it ensures that secondary image understanding can only be executed when the software and hardware conditions are met, avoiding invalid calls or system abnormalities caused by unmet conditions; when any condition is not met, the visual understanding process is automatically skipped and the basic text reply process continues to be executed, so that the multimodal understanding capability and the basic function form an "enhancement" rather than a "dependency" relationship.

[0040] In this embodiment, conservative region cropping of the chat screenshot is performed as follows: the width and height of the chat screenshot are obtained, and the cropping frame is determined according to a preset boundary ratio. For example, the cropping frame is determined according to the ratio that the left boundary is 0.25 of the image width, the top boundary is 0.35 of the image height, the right boundary is 0.98 of the image width, and the bottom boundary is 0.88 of the image height. After boundary correction of the cropping frame coordinates, the cropping is performed to obtain a cropped image of the image evidence that covers the lower part of the chat text and avoids the top title, the left conversation list, and the bottom input box. This avoids irrelevant visual elements such as the top title, the left conversation list, and the bottom input box from entering the visual model and reduces the interference noise of visual interpretation.

[0041] Step S2: Call the visual model to perform structured visual interpretation of the cropped image evidence, clean the interpretation results, and generate image evidence with confidence labels.

[0042] In this embodiment, the visual model is invoked to perform structured visual interpretation of the cropped image evidence as follows: the cropped image evidence and the source clues for the image placement are sent to the visual model. The visual model outputs a structured interpretation result that includes image type, summary, list of key facts, list of keywords, confidence level, and whether user confirmation is required. The image type is selected from one of the following: error screenshot, table, contract, invoice, flowchart, ordinary image, and unknown type. By requiring the visual model to output structured results instead of free text, the uncertainty of the visual model's output is eliminated, enabling subsequent cleaning, evidence construction, and response control stages to be processed based on a unified data format. The categorized output of image types facilitates the implementation of differentiated subsequent processing strategies based on image type (such as generating document verification prompts for contract and invoice images), enhancing the relevance of the response. The output of confidence level and user confirmation mark provides a quantitative basis for subsequent evidence classification and response strategy selection.

[0043] Figure 3 This is a flowchart illustrating the process of cleaning the interpretation results in a desktop chat multimodal response method based on image placeholder positioning and secondary visual interpretation according to an exemplary embodiment of the present invention. Figure 3As shown, in this embodiment, the interpretation results are cleaned in the following ways: The image type is checked to see if it belongs to a preset type set; if not, it is set to an unknown type, preventing abnormal types output by the visual model from polluting subsequent processing and ensuring the standardization of the type field in the evidence object; the confidence level is normalized to the range of 0 to 1, making the confidence level outputs of different visual model versions or under different calling conditions comparable, providing a basis for the unified setting of confidence level thresholds and the stable execution of hierarchical strategies; the keyword list and key fact list are respectively de-emptied, deduplicated, and have their number limited to avoid null values ​​or excessive redundant information from entering the image evidence, reducing the token consumption for subsequent retrieval queries and response generation prompts, and improving efficiency; when the summary is empty, it is replaced with a conservative summary content to avoid the loss of evidence information caused by an empty summary and maintain the integrity of the image evidence.

[0044] Figure 4 A flowchart illustrating the process of generating image evidence with confidence markers according to an exemplary embodiment of the present invention, based on image placeholder localization and secondary visual interpretation, for desktop chat multimodal response, is shown below. Figure 4 As shown, in this embodiment, image evidence with confidence level markers is generated in the following manner: An image evidence object is constructed based on the cleaned interpretation results. This image evidence object includes at least the image type, summary, confidence level, cropped image path, source clues, health level, health description, keywords, key facts, user confirmation marker, and action prompt. When the confidence level is lower than a preset low confidence threshold, the user confirmation marker is forced to indicate "confirmation required," the health level is set to low health, and the health description limits the image evidence to only be used for follow-up confirmation. When the confidence level is not lower than the preset low confidence threshold, the user confirmation marker is set to "confirmation not required," the health level is set to "weak evidence," and the health description indicates that the image evidence needs to be verified in conjunction with the chat context and knowledge base.

[0045] Step S3: Add keywords from the image evidence to the knowledge retrieval query to expand the retrieval scope, and inject the image evidence as weak evidence along with preset usage restrictions into the response to generate prompts.

[0046] As an optional example, in this embodiment, keywords from the image evidence are added to the knowledge retrieval query to expand the retrieval recall scope in the following manner: the original user question is retained, and the keywords are appended to the original user question according to the deduplication rule to form an expanded retrieval query, which improves the retrieval recall effect of low-information questions such as "how to handle this", so that the knowledge base can hit business rules and knowledge entries related to the image content; the keywords are only used to expand the retrieval scope and do not indicate that the image content has been strongly confirmed, strictly distinguishing between the two different technical links of "retrieval enhancement" and "fact confirmation", avoiding the image keywords being mistakenly entered into the knowledge base as confirmed information, and maintaining the consistency and purity of the knowledge base data; and the image summary and key facts are not directly written into the knowledge base as confirmed facts to prevent the spread of erroneous knowledge due to the deviation of image interpretation.

[0047] In this embodiment, the preset usage restrictions include: image evidence is weak evidence; image evidence with a confidence level below a preset low confidence threshold can only be used for follow-up questioning and confirmation; follow-up questioning takes precedence when image evidence conflicts with knowledge base retrieval results; generating any factual data, entity identifiers, or subjective conventions based on image evidence without structured interpretation support is prohibited; when the image type or keywords of the image evidence indicate the existence of structured documents or voucher information that needs to be verified (such as contracts, invoices, forms, attachments, or files), an action prompt is generated to guide the user to verify the source of the external document.

[0048] This embodiment establishes a complete set of guidelines for using image evidence at the response generation prompt level through multiple usage restrictions. This allows the generation model to utilize image information under clear constraints, effectively preventing the model from "over-interpreting" or "arbitrarily inferring" sensitive information not present in the image. It explicitly prohibits the generation of any factual data, entity identifiers, or subjective agreements based on image evidence without structured interpretation support, thus eliminating the risk of erroneous output in sensitive scenarios involving financial amounts, dates and times, serial numbers, codes, and contractual commitments.

[0049] In practical applications, this embodiment also includes a degradation mechanism for failures at any stage. For example, if any stage from S1 to S3 fails, the use of image evidence is terminated, the original text recognition, knowledge retrieval, and response generation process is executed, and a prompt for supplementary original image or manual confirmation is output. This ensures that failures at any stage, such as unreadable screenshots, failed cropping, visual model timeouts, abnormal returns, or invalid results, will not interrupt the entire response process, guaranteeing basic text response services even when some capabilities are unavailable. The prompt for supplementary original image or manual confirmation in case of failure informs the user that the current response does not include image information, guiding the user to supplement with a clear original image or for manual intervention, ensuring the integrity and transparency of the user experience. The "degradation rather than crash" design in abnormal states makes multimodal understanding capabilities a true "enhancement layer" rather than a "replacement layer" for basic functions, reducing the complexity of operation and maintenance and troubleshooting, and enhancing the feasibility of the solution in actual engineering deployments.

[0050] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0051] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation, characterized in that, The method includes Step S1: In response to reading an image attachment placeholder from the chat context and meeting the triggering conditions, perform conservative region cropping on the chat screenshot to obtain a cropped image evidence; Step S2: Call the visual model to perform structured visual interpretation of the cropped image evidence, clean the interpretation results, and generate image evidence with confidence labels; Step S3: Add keywords from the image evidence to the knowledge retrieval query to expand the retrieval scope, and inject the image evidence as weak evidence along with preset usage restrictions into the response to generate prompts.

2. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S1, image attachment placeholders are read from the chat context in the following way: non-text messages in the chat messages are converted into placeholder text through chat screenshot recognition, and the placeholder type, placeholder tag, message number, speaker and placeholder description are recorded as attachment placeholder clues. Among them, image placeholders include at least image placeholders and screenshot placeholders.

3. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S1, the triggering conditions include: there is an image placeholder in a preset number of messages within a preset time range, the current screenshot file is readable, and the visual model configuration is valid.

4. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S1, conservative region cropping is performed on the chat screenshot as follows: obtain the image width and height of the chat screenshot, determine the cropping frame according to the preset boundary ratio, and perform cropping after boundary correction of the cropping frame coordinates to obtain a cropped image of the evidence that covers the lower part of the chat text and avoids the top title, left conversation list and bottom input box.

5. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S2, the visual model is invoked to perform a structured visual interpretation of the cropped image evidence as follows: The cropped image evidence and the source clues of the image placeholder are sent to the visual model. The visual model outputs a structured interpretation result that includes the image type, summary, list of key facts, list of keywords, confidence level, and whether user confirmation is required. The image type is selected from one of the following: error screenshot, table, contract, invoice, flowchart, ordinary image, and unknown type.

6. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S2, the interpretation results are cleaned as follows: the image type is checked to see if it belongs to the preset type set; if not, it is set to unknown type; the confidence level is normalized to the range of 0 to 1; the keyword list and the key fact list are removed from empty, duplicates are removed, and the number of entries is limited; when the summary is empty, it is replaced with conservative summary content.

7. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S2, image evidence with confidence level markers is generated as follows: An image evidence object is constructed based on the cleaned interpretation results. This image evidence object includes at least the image type, summary, confidence level, cropped image path, source clues, health level, health description, keywords, key facts, user confirmation marker, and action prompt. When the confidence level is lower than a preset low confidence threshold, the user confirmation marker is forced to indicate "confirmation required," the health level is set to low health, and the health description limits the image evidence to only be used for follow-up confirmation. When the confidence level is not lower than the preset low confidence threshold, the user confirmation marker is set to "confirmation not required," the health level is set to "weak evidence," and the health description indicates that the image evidence needs to be verified in conjunction with the chat context and knowledge base.

8. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S3, keywords from the image evidence are added to the knowledge retrieval query to expand the retrieval scope in the following way: the original user question is retained, and the keywords are appended to the original user question according to the deduplication rules to form an expanded retrieval query; these keywords are only used to expand the retrieval scope and do not indicate that the image content has been strongly confirmed, and the image summary and key facts are not directly written into the knowledge base as confirmed facts.

9. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, In step S3, the preset usage restrictions include: image evidence is weak evidence; image evidence with a confidence level lower than the preset low confidence threshold can only be used for follow-up questioning and confirmation; follow-up questioning is given priority when image evidence conflicts with the knowledge base retrieval results; generating any factual data, entity identifiers, or subjective conventions based on image evidence without structured interpretation is prohibited; when the image type or keywords of the image evidence indicate the existence of structured documents or voucher information that need to be verified, an action prompt is generated to guide the user to verify the source of the external document.

10. The desktop chat multimodal reply method based on image placeholder positioning and secondary visual interpretation according to claim 1, characterized in that, If any step from S1 to S3 fails, the use of image evidence is terminated, the original text recognition, knowledge retrieval and response generation process is executed, and a prompt message for supplementing the original image or requiring manual confirmation is output.

Images