A customer service automatic reply anti-serial sending and anti-repetition method and system based on desktop visual recognition

Abstract

The application discloses a customer service automatic reply anti-serial sending and anti-repetition method and system based on desktop visual identification. The method is applied to a local customer service node running a target reception software desktop client. When a normal message body is acquired, local chat history or local session record is preferentially read. When local reading is unavailable, erroneous or needs to be rolled back according to a program strategy, a current visible message is recognized through desktop window screenshot OCR, and an auxiliary function control tree text is used for current chat title confirmation, window or control state auxiliary judgment and diagnostic information acquisition. The system recognizes customer side messages based on window layout parameters and filters time stamps, interface control texts and customer service side sent messages. Message keys are respectively generated for OCR source messages and local history source messages, and combined with a cooling queue and a 300-second latest sending segment for repetition removal and self-reply filtering. After local knowledge base retrieval, reply generation and strategy auditing, the method is written into a to-be-sent queue, and then the current session object is verified again through OCR title confirmation and auxiliary function control tree before sending, and only when matching, text or pictures are sent. The application can reduce the risk of serial sending, repeated sending and self-reply cycle in the desktop customer service automation scene.

Description

Technical Field

[0001] This invention relates to the fields of desktop client automated processing, computer vision recognition, intelligent customer service, and local message processing, and in particular to a method and system for preventing duplicate and unauthorized automatic customer service replies based on desktop visual recognition. Background Technology

[0002] Existing intelligent customer service systems typically access customer messages and generate responses through platform open interfaces, web-based customer service plugins, or manual copy-and-paste methods. In some desktop client scenarios, business systems may not be able to use official customer service interfaces, or due to considerations such as deployment costs, account permissions, data retention, and on-site operating conditions, message retrieval and sending operations need to be completed on the desktop window of the target customer service software.

[0003] In these non-standard interface scenarios, common solutions involve manual screen monitoring and responses, or using a simple script to perform OCR recognition on a screenshot of the desktop window and then sending a fixed script. However, the desktop window layout changes with resolution, window width, conversation list status, search overlay, temporary conversations, and input area. Simple OCR can easily mix the left-side conversation list, timestamps, system buttons, old messages, or interface text into the customer's question.

[0004] Furthermore, the chat bubble sent by the customer service representative themselves might also be recognized by OCR as a new customer message, triggering a loop around their own reply and causing duplicate sending. If the current chat recipient is not confirmed before sending, if the window is accidentally clicked on to another customer, the search overlay is obscured, or the current conversation title fails to be recognized, a customer's reply might be sent to another customer instead.

[0005] Furthermore, relying solely on generative models can easily lead to issues such as low confidence, refusal to answer, identity-revealing statements, or inconsistencies with business data; while relying solely on fixed knowledge bases struggles to handle continuous follow-up questions, image data, and event data. During long-term background monitoring, if faults such as abnormal permissions, invisible windows, failed submissions, database busy locks, and log bloat are not visible, they can easily manifest as the program still running but not actually responding.

[0006] Therefore, it is necessary to provide an automatic reply method and system suitable for desktop non-standard interface customer service scenarios, so that it can complete message recognition, deduplication, knowledge base integration, policy review, pre-send session confirmation and failure handling locally, thereby reducing the risks of cross-sending, duplicate sending and self-reply loops. Summary of the Invention

[0007] This invention aims to solve the problems of erroneous replies, cross-sending, duplicate sending, and long-term invisibility caused by existing desktop customer service automation solutions in scenarios where there is no stable platform interface or it is not suitable to upload session data. These problems arise from OCR misidentification, changes in window layout, misjudgment of customer service bubble, lack of confirmation of the session object before sending, and unstable model responses.

[0008] The focus of this invention is not simply to use a large model to generate customer service scripts, but to combine local session reading, desktop visual recognition, window area classification, message key deduplication, recently sent segment filtering, pending send queue binding, and pre-send session confirmation into a closed-loop processing flow to improve the secure and reliable sending of automatic customer service nodes on the desktop.

[0009] To address the aforementioned issues, this invention provides a method for preventing duplicate and repetitive automatic customer service replies based on desktop visual recognition, applied to a local customer service node running the target reception software's desktop client. This method first acquires the target reception software's local session information and current desktop window information. Local session information may include local chat history or local session records; the current desktop window information may include the OCR recognition result of a window screenshot and the text of the accessibility control tree.

[0010] During the message retrieval phase, the local customer service node prioritizes determining the body of ordinary messages based on local chat history or local session records. When local retrieval is unavailable, encounters an error, or requires rollback according to program policy, the local customer service node determines the currently visible ordinary message body based on the OCR recognition result of the window screenshot. The auxiliary function control tree text is mainly used for confirming the current chat title, assisting in judging window or control status, and obtaining diagnostic information, and forms a dual-path verification with the OCR title confirmation before sending.

[0011] After obtaining candidate messages, the local customer service node performs region classification based on the message body, OCR text box coordinates, confidence level, and window area information, according to the current window layout parameters. Region classification is used to distinguish between customer-side messages, customer service-side sent messages, timestamps, chat titles, left-side conversation lists, and UI control text, thereby reducing the risk of misidentifying non-customer issues as messages awaiting a response.

[0012] For identified candidate customer messages, the local customer service node generates message keys based on different message key generation rules according to the message source. For visible messages from OCR sources, visible message keys can be generated from the current session title, normalized text summary, vertical coordinate binning, and horizontal coordinate features, and the processing time is recorded for validity management. For messages from local historical sources, historical message keys can be generated from the customer identifier or session identifier, message timestamp, and normalized text summary. By combining session identifiers, text fingerprints, visual location fingerprints, and time information, the risk of duplicate processing can be reduced.

[0013] After identifying the customer message to be responded to, the local customer service node searches its local knowledge base and generates a draft response based on the search results. The knowledge base can include formal business materials, image materials, and event materials. For complete business questions, the current customer message to be responded to is preferred for retrieval; for short follow-up questions, the most recent valid context can be used for retrieval. After the draft response is generated, it enters a policy review process. This review checks for sensitive words, low confidence levels, uncertain wording, wording that exposes the bot's identity, and risks related to missing information.

[0014] When the policy review allows automatic sending, the local customer service node first writes the target customer or target session, message key, customer message to be replied to, reply content, confidence level, and status into the sending queue. Before sending, the session confirmation module recognizes the current chat title using OCR; if the OCR title recognition fails, is empty, has insufficient confidence, or the recognition result does not match the target customer or target session, it reads the auxiliary function control tree text in the top area of ​​the window for secondary confirmation. The sending execution module only executes text or image sending if the current session object matches the target customer or target session in the sending queue; otherwise, it cancels the sending and records the reason for failure.

[0015] This invention also provides a desktop-based visual recognition-based automatic customer service reply system to prevent duplicate and repetitive messages. The system includes a local session reading module, a desktop visual execution module, a message recognition module, a message key and deduplication module, a knowledge base module, a reply generation and strategy review module, a pending message queue module, a session confirmation module, a sending execution module, and a failure handling module. These modules work together to implement the aforementioned method.

[0016] Beneficial effects: First, by prioritizing local session reading, using OCR rollback recognition, and assigning tasks to the auxiliary function control tree title / status confirmation, the reliability of recognizing ordinary message text and the current session object in desktop non-standard interface customer service scenarios can be improved.

[0017] Secondly, by using window layout parameters, candidate message coordinates, and regional classification to distinguish between customer-side messages, customer service-side sent messages, timestamps, and UI control text, the risk of simple OCR mixing non-customer questions into messages awaiting a reply can be reduced.

[0018] Third, by generating message keys for OCR source messages and local historical source messages respectively, and combining them with the processed set, cooling queue and the list of most recently sent fragments in 300 seconds, the risk of the same customer message being processed repeatedly and the customer service representative's own reply being identified as a new customer problem can be reduced.

[0019] Fourth, by binding the target customer or target session, original question, reply content and status to the queue to be sent before sending, and confirming the current session object through OCR title and auxiliary function control tree text before sending, the risk of cross-sending caused by window switching error, search overlay obstruction or temporary session abnormality can be reduced.

[0020] Fifth, by combining local knowledge base retrieval, model generation, standard answer fallback, and strategy review, it is possible to handle short follow-up questions, activity data, and image data while ensuring the consistency of business data, thereby reducing the risk of the model responding with low confidence or refusing to answer even when data is available.

[0021] Sixth, by using failure queues, status recording, manual processing entry points, single instance locks, running heartbeats, health checks, log rotation, diagnostic export, and guardian restart, the stability and fault visibility of local customer service nodes during long-term operation can be improved.

[0022] Seventh, by running locally and isolating the data directory, model service credentials, customer sessions, customer profiles, and diagnostic information can be stored locally by default on the customer service node, reducing data leakage caused by accessing the cloud customer service system. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the system composition and connection relationship provided for an embodiment of the present invention.

[0024] Figure 2 A message processing flowchart provided for embodiments of the present invention.

[0025] Figure 3 This is a flowchart illustrating the anti-scraping and anti-duplication processing steps provided in an embodiment of the present invention.

[0026] Figure 4 A flowchart illustrating the knowledge base fusion response process provided in this embodiment of the invention.

[0027] Figure 5 The flowchart illustrates the failure handling and local protection process provided in this embodiment of the invention. Detailed Implementation

[0028] The embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the described embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. The following embodiments and features thereof can be combined with each other without contradiction.

[0029] The target reception software referred to in this specification refers to desktop client software used to receive customer messages, including but not limited to instant messaging software, enterprise communication software, or other reception software that displays customer conversations in the form of desktop windows. The local customer service node refers to a computing device running on the customer service personnel's local terminal or local server. The target desktop operating system can be macOS; other desktop operating systems can also be used if equivalent screenshot, OCR, window control reading, and input control capabilities are available.

[0030] Example 1: System Composition like Figure 1 As shown, the system in this embodiment includes a local session reading module, a desktop visual execution module, a message recognition module, a message key and deduplication module, a knowledge base module, a response generation and policy review module, a pending send queue module, a session confirmation module, a send execution module, and a failure handling module.

[0031] The local session reading module is used to prioritize reading the chat history information or local session records generated locally by the target reception software. The local chat history information or local session records may include fields such as customer identifier or session identifier, message timestamp, message text, and message direction.

[0032] The Desktop Vision Execution Module is used to activate the target reception software desktop window and perform actions such as screenshotting, OCR recognition, clicking, pasting, text sending, image sending, and accessibility control tree reading. On macOS, the Desktop Vision Execution Module can call Vision OCR to perform text recognition, call AppKit or equivalent system interfaces to obtain window screenshots, and call the Accessibility interface to read the window's top title area, control states, and diagnostic-related text.

[0033] The message recognition module is connected to the local session reading module and the desktop visual execution module, and is used to receive local chat history information, OCR text boxes, screenshots, and auxiliary function text. In this embodiment, the body of ordinary messages is mainly determined by local chat history information or OCR recognition results, while the auxiliary function control tree text is used for current chat title confirmation, window or control status assistance judgment, and diagnostic information acquisition. The message recognition module is also used to identify client-side messages based on window layout parameters and filter session list text, timestamps, system buttons, input area text, and messages sent by the client.

[0034] The message key and deduplication module is used to generate message keys for different message sources. For OCR-sourced messages, the module can generate visible message keys based on the current session title, text normalization summary, vertical coordinate binning, and horizontal coordinate features, and record the processing time. For local historical source messages, the module can generate historical message keys based on the client identifier or session identifier, message timestamp, and text normalization summary.

[0035] The knowledge base module is used to import official documents, image data, and event information, and to create a search index. Official documents can include business policies, product descriptions, pricing information, after-sales rules, or standard Q&As; image data can include product images, event posters, QR codes, screenshots of instructions, or after-sales voucher templates; event information can include rules for limited-time events, preferential conditions, registration methods, or inventory status descriptions.

[0036] The response generation and strategy review module is used to generate draft responses based on customer messages and knowledge base search results, and then review these draft responses. Review dimensions may include sensitive words, low confidence levels, uncertain wording, wording that exposes a robot's identity, risks of missing information, and risks of inconsistencies with the business knowledge base.

[0037] The pending-to-send queue module is used to create pending-to-send records before sending. Each pending-to-send record includes at least the target customer or target session, message key, original customer question, reply content, trigger reason, confidence level, status, and failure reason. The status can include pending, sent, failed, retry, manual, or terminal failed.

[0038] The session confirmation module is used to confirm the current session object before sending. The module first identifies the current chat title using OCR; if the OCR title recognition result is empty, has low confidence, or does not match the target customer or target session bound to the record to be sent, it performs a secondary confirmation using the title text in the top area of ​​the accessibility control tree text reading window. If both OCR title confirmation and accessibility control tree text confirmation fail or both do not match, the session confirmation module outputs a "send prohibited" result.

[0039] The send execution module is used to paste the reply content into the input box of the target reception software and send it after the session is confirmed, or to send an image based on the image path in the reply content. The failure handling module is used to receive exceptions at each stage, record the reasons for failure, and provide entry points for retrying, manual handling, or termination of failure handling.

[0040] Example 2: Message Acquisition and Region Classification like Figure 2As shown, after the local customer service node starts, it loads the configuration file, knowledge base, node identity, session database, and message status, and acquires a single-instance lock. Acquiring a single-instance lock can prevent multiple automation processes on the same customer service node from operating the same desktop window simultaneously.

[0041] During the message acquisition phase, the local customer service node prioritizes reading the chat history or local session records generated locally by the target reception software. If local reading is unavailable, fails, lacks sufficient permissions, or requires rollback according to program policy, the desktop vision execution module takes a screenshot of the current desktop window and performs OCR recognition to identify the currently visible ordinary message text. The accessibility control tree text is used at this stage for confirming the current chat title, assisting in judging window or control status, and obtaining diagnostic information.

[0042] Local chat history or local session records can output customer or session identifiers, message timestamps, message direction, and message text. OCR recognition results can output text content, text box coordinates, text box size, and confidence level. Accessibility control tree text can output window title, top control text, button text, input box status, session list status, and other diagnostic text. The message recognition module processes the above information to form a candidate message set and a candidate current session title set.

[0043] During the region classification phase, the message recognition module selects a corresponding threshold from a preset window layout threshold group based on the current window width, the boundary of the conversation list region, the boundary of the chat body region, the boundary of the input region, and the boundary of the chat title region. The preset window layout threshold group can be configured according to window width levels or the layout type of the target reception software.

[0044] For each candidate OCR text box, the message recognition module can determine whether the candidate OCR text box belongs to the customer message area, the customer service sent message area, the chat title area, the timestamp area, the conversation list area, or the interface control area based on the horizontal position, vertical position, height, width, and relative relationship of the center point of the text box to the chat body area.

[0045] When candidate text is located in the conversation list area, system button area, search box area, input area, or timestamp area, the message recognition module filters the candidate text. When candidate text is located in the customer message area and is not a processed message, is not in a cooldown period, and does not match the most recently sent segment, the message recognition module treats it as a customer message to be replied to.

[0046] Example 3: Message Key Deduplication and Self-Reply Filtering like Figure 3 As shown, this embodiment uses message keys, processed sets, cooling queues, and recently sent fragment lists to jointly prevent duplicates and self-replies.

[0047] For each candidate customer message, the message recognition module first performs text normalization. Normalization may include removing redundant whitespace, merging line breaks, standardizing full-width and half-width punctuation, deleting semantically unrelated interface prefixes, and normalizing case.

[0048] After normalization, the local customer service node generates message keys according to the candidate message source. For visible messages originating from screenshot OCR, the local customer service node can combine the current session title, normalized text digest, vertical coordinate binning, and horizontal coordinate features, and perform hashing to obtain the visible message key. Simultaneously, the processing time is recorded in the message status for validity management. Vertical coordinate binning reflects the approximate vertical position of the message in the chat body area, while horizontal coordinate features indicate whether the message is located in the customer-side bubble area or the customer service-side bubble area.

[0049] For historical messages originating from local chat history or local session records, the local customer service node can combine the customer identifier or session identifier, message timestamp, and normalized text digest, and then perform hashing to obtain the historical message key. Through these different message key rules, the system can simultaneously utilize local session identifiers, text fingerprints, visual location fingerprints, and time information to achieve anti-duplicate processing.

[0050] If the message key already exists in the processed set, it means the corresponding customer message has been replied to or manually processed, and the local customer service node ignores the candidate message. If the message key is within the preset cooldown time range, it means the same message may be recognized again due to window refresh, repeated OCR recognition, or scrolling jitter, and the local customer service node also ignores the candidate message.

[0051] For messages already sent by the customer service representative, the local customer service node records the most recently sent text after successful transmission. This most recently sent text can be normalized and broken down into several recently sent segments, which are then stored in association with the sending time. In one specific embodiment, the valid time for the most recently sent segment is determined by a preset parameter within the program to be 300 seconds. If a subsequent candidate message is located within the area of ​​messages already sent by the customer service representative, or if the candidate message text matches a recently sent segment within the 300-second valid timeframe, then the candidate message is considered a message already sent by the customer service representative, and no further response is triggered.

[0052] Example 4: Knowledge Base Integration, Response Generation, and Policy Review like Figure 4 As shown, the knowledge base module in this embodiment can build a search index from official documents, image data, and event data. Official documents are used to provide stable business answers; image data are used to provide image paths or image identifiers when images need to be sent; and event data are used to handle limited-time events, discount rules, registration processes, or temporary notices.

[0053] When a customer message is a complete business question, the knowledge base module preferentially uses only currently pending customer messages for retrieval to avoid contaminating the current question with historical responses. When a customer message is a short follow-up question, such as "Are there any more?", "How do I participate?", or "How much does this cost?", the knowledge base module can combine the most recent valid context for retrieval. The most recent valid context can come from recent customer messages to the same target customer and answers already sent by the system, but does not include messages judged as invalid, failed, or manually taken over.

[0054] The response generation and strategy review module constructs model suggestions based on the search results and calls the model service to generate a draft response. The model service can be deployed locally or is an external model service called through a local node; regardless of the deployment method, target customer sessions, customer profiles, model service credentials, and diagnostic information are preferably stored in a local data directory in isolation.

[0055] The strategy review module reviews the draft responses generated by the model. Review items include, but are not limited to: whether it contains sensitive words, whether it uses language that is not suitable for revealing one's identity such as "I am a robot" or "I am an AI model", whether it uses language that refuses to answer or has low confidence such as "I don't know" or "I cannot answer", whether it lacks supporting business information, whether it is inconsistent with the content strongly matched in the knowledge base, and whether it requires sending images but the image paths are missing.

[0056] When the knowledge base contains strong hit results but the model generates a rejection or unanswerable result, the response generation and strategy review module can extract a standard answer from the strong hit results as a fallback response and re-review the strategy. If the re-review still fails, the local customer service node will not execute automatic sending, but will instead change the corresponding pending record to a pending manual processing status.

[0057] Example 5: Pending Send Queue, Session Confirmation, and Send Execution like Figure 3 As shown, in this embodiment, data is first written into a queue to be sent before execution. Each record in the queue is bound to the target customer or target session, the original customer question, the message key, the reply content, the trigger reason, the review result, the confidence level, and the current status. By enqueuing first and then confirming, each automatic send can have a traceable target customer or target session and the original trigger message.

[0058] Before sending, the session confirmation module performs OCR recognition on the chat title area of ​​the target reception software's current desktop window to obtain the current session title. If the OCR title recognition fails, the recognition result is empty, the confidence level is lower than the preset threshold, or the recognition result does not match the target customer or target session bound to the record in the queue to be sent, the session confirmation module further traverses the accessibility control tree text in the top area of ​​the window to extract possible current session titles.

[0059] The session confirmation module matches the current session title with the target customer or session in the pending delivery queue. Matching can be done using exact match, normalized match, alias mapping match, or match based on customer ID. If the match is successful, the delivery module pastes the response content into the input box and triggers delivery, or sends an image based on the image path. If neither the OCR title nor the accessibility control tree text can confirm that the current session object is the target customer or session, or if the match fails, the current session title is empty, the search overlay is obscured, the window is invisible, or the target reception software is not operational, the session confirmation module outputs a "delay delivery" message and throws or records an error.

[0060] Upon successful transmission, the pending transmission queue module sets the record status to "sent" and adds the transmitted content to the recently transmitted segment list. If transmission fails, the failure handling module records the reason for the failure and, based on the exception type, sets the record to "retry," "manual," or "terminal failed."

[0061] Through the above processing, even if the desktop window is accidentally clicked on another customer, the chat title is obscured, the temporary session cannot be searched, the OCR recognition fails, or the accessibility control tree cannot confirm the title, the local customer service node will not automatically send without confirming the current session object, thereby reducing the risk of cross-sending.

[0062] Example 6: Failure Handling, Local Guardianship, and Data Isolation like Figure 5 As shown, the failure handling module in this embodiment receives exceptions from each stage of message acquisition, message recognition, knowledge base retrieval, response generation, policy review, session confirmation, and execution. Exceptions may include: the target reception software window being invisible, accessibility permission errors, OCR recognition failure, database busy lock, missing knowledge base, model call failure, pasting failure, sending failure, invalid image path, and log writing failure.

[0063] The failure handling module records the reason for each failure, the time of occurrence, the target customer or target session, the message key, the current window state, and optional diagnostic information. Based on the exception type and retry policy, the failure handling module moves the pending record to a state of pending retry, pending manual processing, or terminated failure.

[0064] To enhance long-term operational capability, local customer service nodes can be configured with single-instance locks, running heartbeats, health checks, log rotation, diagnostic export, and guardian restart mechanisms. Single-instance locks prevent repeated processes from operating on the same desktop window; running heartbeats indicate node liveness; health checks check window visibility, permission status, database availability, and knowledge base status; log rotation prevents log file bloat; diagnostic export is used for on-site troubleshooting; and guardian restart restarts the node after an abnormal exit.

[0065] Local customer service nodes can have independent data directories set up for different customer service accounts or different customer service software instances. The data directory is used to store node configurations, model service credentials, knowledge base indexes, chat status, pending messages queues, recently sent segments, failure records, and diagnostic logs. Data directory isolation reduces state contamination between different customer service accounts and ensures that customer sessions and diagnostic information are stored locally by default.

[0066] The modules in the above embodiments can be implemented by software programs or by a combination of software and general-purpose hardware. The division of modules is for the convenience of describing functions and does not limit the physical deployment method; in actual implementation, multiple modules can be integrated into the same process, the same service, or the same electronic device.

Claims

1. A method for preventing duplicate and repetitive messages in automated customer service replies based on desktop visual recognition, characterized in that, This method, applied to a local customer service node running a target reception software desktop client, includes: acquiring the current desktop window information and local session information of the target reception software; the local session information includes local chat history information or local session records; the current desktop window information includes the OCR recognition result of a window screenshot and auxiliary function control tree text; preferentially determining the ordinary message body based on the local session information; when the local session information is unavailable, erroneous, or requires rollback according to program strategy, determining the currently visible ordinary message body based on the OCR recognition result of the window screenshot, and using the auxiliary function control tree text for current chat title confirmation, window or control state auxiliary judgment, and diagnostic information acquisition; determining candidate messages based on at least one of the ordinary message body, OCR text box coordinates, confidence level, and window area information, and classifying the candidate messages by region according to the current window layout parameters to distinguish... The system categorizes messages into customer-side messages, customer service-side sent messages, timestamps, and UI control text. Message keys are generated using different rules based on the candidate message source, and duplicate messages and messages sent by the customer service representative are filtered based on processed message keys, a cooling-off queue, and recently sent segments. After identifying a customer message to be replied to, a local knowledge base is retrieved and a draft reply is generated. This draft reply undergoes policy review. When the policy review allows automatic sending, the target customer or target session, message key, customer message to be replied to, and reply content are first written into the sending queue. Before sending, the current session object is confirmed, including OCR title confirmation and secondary confirmation based on the auxiliary function control tree text when OCR cannot confirm or the confirmation does not match. Sending is only performed when the current session object matches the target customer or target session bound in the sending queue, and the status of successful sending, failed sending, or pending manual processing is recorded.

2. The method according to claim 1, characterized in that, The acquisition of the target reception software's current desktop window information and local session information includes: reading the chat history information or local session records generated locally by the target reception software to obtain the customer identifier or session identifier, message timestamp, and message text; when the chat history information or local session records are unavailable, reading fails, permissions are insufficient, or rollback is required according to the program strategy, a screenshot of the target reception software's current desktop window is taken and OCR recognition is performed to obtain the text of the currently visible message, text box coordinates, text box size, and confidence level; the auxiliary function control tree text corresponding to the current desktop window is traversed to obtain the window title, top control text, current control status, and abnormal diagnosis text, wherein the ordinary message body is determined by the local session information or the OCR recognition result.

3. The method according to claim 1, characterized in that, The process of classifying candidate messages by region according to the current window layout parameters includes: selecting a corresponding threshold from a preset window layout threshold group based on the current window width, chat text area boundary, conversation list area boundary, input area boundary, and chat title area boundary; dividing the candidate message text box into customer message area, customer service sent message area, chat title area, timestamp area, and interface control area based on the horizontal position, vertical position, and size of the candidate message text box; and filtering text located in the conversation list area, system button area, search box area, input area, or timestamp area.

4. The method according to claim 1, characterized in that, The process of generating message keys based on different message key generation rules according to the source of candidate messages includes: for visible messages originating from window screenshot OCR recognition results, generating visible message keys based on the current session title, normalized summary of candidate message text, vertical coordinate binning, and horizontal coordinate features, and recording the processing time for validity management; for historical messages originating from local chat history information or local session records, generating historical message keys based on customer identifier or session identifier, message timestamp, and normalized summary of candidate message text; wherein, the local customer service node achieves anti-duplicate processing through session identifier, text fingerprint, visual location fingerprint, and time information.

5. The method according to claim 1, characterized in that, The most recently sent segment is formed by normalizing and segmenting the successfully sent reply text, and is stored in association with the sending time; the most recently sent segment has a valid time determined by preset parameters in the program, and in a specific embodiment, the valid time is 300 seconds; when the candidate message is located in the area of ​​messages already sent by the customer service, or when the candidate message text matches the most recently sent segment within the valid time, the candidate message is determined to be a message already sent by the customer service itself, so as to avoid triggering automatic replies in a loop around its own replies.

6. The method according to claim 1, characterized in that, The process of retrieving the local knowledge base and generating a draft response includes: creating or calling a search index from official documents, image data, and event data; for complete business questions, using the current customer message to be replied to for retrieval; for short follow-up questions, using the current customer message to be replied to and the most recent valid context for retrieval; constructing model prompts based on the search results and generating a draft response, which includes a text reply and / or the path to the image to be sent.

7. The method according to claim 6, characterized in that, The strategy review of the draft response includes: checking whether the draft response contains sensitive words, low confidence, uncertain wording, wording that exposes the robot's identity, or lacks information; when the local knowledge base contains strong hit results but the model generates a rejection or inability to answer, extract the standard answer from the strong hit results and re-review the strategy; if the re-review still fails, transfer the corresponding message to the pending manual processing state.

8. The method according to claim 1, characterized in that, The step of confirming the current session object before sending includes: performing OCR title recognition from the chat title area of ​​the current desktop window of the target reception software; when the OCR title recognition result is empty, the confidence level is lower than a preset threshold, or the OCR title recognition result does not match the target customer or target session bound in the queue to be sent, traversing the auxiliary function control tree text in the top area of ​​the window to obtain the current session title; matching the obtained current session title with the target customer or target session bound in the queue to be sent; when both OCR title confirmation and auxiliary function control tree text confirmation cannot confirm a match, canceling the sending and recording the error reason.

9. The method according to claim 1, characterized in that, The record sending success, sending failure, or pending manual processing status includes: setting the record in the sending queue to one of the following states: pending, sent, failed, retry, manual, or terminal failed; recording the reason for failure for at least one of the following exceptions: window not visible, permission error, pasting failure, sending failure, database busy lock, knowledge base missing, invalid image path, and model call failure; and maintaining the continuous operation of the local customer service node through at least one of the following mechanisms: single instance lock, running heartbeat, health check, log rotation, diagnostic export, and guardian restart.

10. A desktop-based visual recognition-based automatic customer service reply system to prevent duplicate and repetitive messages, characterized in that, include: The local session reading module is configured to prioritize reading the chat history or local session records generated locally by the target client software. The desktop visual execution module is configured to activate the target reception software desktop window, perform screenshot, OCR recognition, click, paste, text sending and / or image sending, and read the auxiliary function control tree text, which is used for current chat title confirmation, window or control status auxiliary judgment and diagnostic information acquisition; the message recognition module is configured to determine the currently visible ordinary message body based on the OCR recognition result when the local session is unavailable, malfunctions, or needs to be rolled back according to the program strategy, and to identify customer-side messages and filter customer-side sent messages, timestamps, and interface control text according to window layout parameters; the message key and deduplication module is configured to generate message keys for OCR source messages and local historical source messages respectively, and filter duplicate messages and customer-side messages sent by the customer service itself based on processed message keys, cooling queue, and recently sent fragments; the knowledge base module is configured to store formal materials, image materials, and activity materials and provide search results; the reply generation and strategy review module is configured to generate reply drafts and determine whether automatic sending is allowed; The pending-send queue module is configured to record the target customer or target session, message key, customer message, reply content, confidence level, and status before sending; the session confirmation module is configured to verify the current session object through OCR title recognition and secondary confirmation of the auxiliary function control tree text before sending is executed. The sending execution module is configured to send only when the current session object matches the target client or target session in the sending queue; and the failure handling module is configured to record the reason for the exception and transfer the message to a pending retry, pending manual intervention, or termination failure status.

Images