Mouse voice input real-time on-screen and artificial intelligence asynchronous optimization method

Abstract

The application discloses a mouse voice input real-time on-screen and artificial intelligence asynchronous optimization method. When the mouse button is pressed, the client software establishes a voice input session and takes the current focus input control as the target input control, collects the user voice, differentially generates stable incremental text, and writes the stable incremental text into the target input control through a system text interface in a non-blocking manner; when the unoptimized text meets the preset word number, pause or complete sentence condition, the text segment to be optimized is determined and a dynamic position anchor point is established, and the text segment to be optimized is sent to an artificial intelligence optimization module in an asynchronous mode to obtain optimized candidate text without blocking recognition and writing; the optimized floating window is displayed in a non-activated window mode without losing the input focus of the target input control, and the anchor point is offset and corrected with the writing of the stable incremental text; if the identification is consistent and the text verification is passed, the replacement is completed based on the corrected anchor point. The application alleviates the blocking conflict between real-time on-screen and artificial intelligence optimization, and reduces the risk of replacement misplacement.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction, speech recognition, and computer text input technology, specifically to a method, system, and storage medium for real-time on-screen display of mouse voice input and asynchronous optimization with artificial intelligence. Background Technology

[0002] With the rapid development of speech recognition and artificial intelligence generation technologies, voice-based text input is increasingly being applied in scenarios such as office work, instant messaging, and content creation. Users can significantly improve input efficiency by inputting text via voice. Furthermore, by using artificial intelligence models to optimize the speech recognition results (e.g., error correction, polishing, punctuation correction, and style adjustment), the shortcomings of speech recognition in areas such as colloquialism, word filling, and sentence segmentation can be further compensated for, thus improving the quality of the output text.

[0003] At the hardware level, smart mice with voice buttons have been proposed. For example, existing technology (see Chinese invention patent application CN 108509068 A) discloses a smart gaming mouse that uses speech recognition to achieve speech-to-text transcription. This mouse, by setting a voice button, a voice signal acquisition module, a voice signal conversion module, and an online or offline speech recognition engine, allows users to initiate speech-to-text transcription or voice macro commands through mouse buttons. However, this type of solution mainly focuses on the mouse's own hardware structure, the switching of the speech recognition engine, and the definition and management of voice macros. It does not focus on solving the problems of real-time and stable display of speech recognition results in the user's current input control, asynchronous collaboration with the artificial intelligence optimization module, and the anchoring and accurate replacement of the optimized results.

[0004] Regarding the binding of voice input with a target editing area, existing technologies disclose a method that determines the target editing box where the cursor is located on the current page, obtains and stores the identifier of the target editing box, and writes the text obtained from voice recognition into the target editing box according to the identifier after receiving voice input from the user. This type of method can solve the problem of writing voice recognition results into a specified editing box to some extent, but it mainly focuses on the binding relationship between the recognition result and the target editing box. It does not address the process of real-time streaming of voice recognition results, the parallel collaboration between the artificial intelligence optimization module and the real-time streaming process, or the anchoring, verification, and accurate replacement of specific text segments in the already displayed text during continuous user voice input or manual editing.

[0005] Regarding focus positioning during voice input, existing technology (see Chinese invention patent application CN 112181164 A) discloses an intelligent voice typing method based on cursor focus coordinate positioning. This method monitors the cursor focus coordinates of the text box during voice typing, binds the cursor focus to the current text box when voice typing is interrupted by another thread, and determines the starting position of subsequent text input based on the last monitored cursor focus coordinates before the interruption. This type of method primarily addresses the focus recovery problem after voice typing is interrupted by an external thread. It is a passive handling mechanism for abnormal input focus and cannot directly solve the problems of text segment position drift and accurate replacement when real-time on-screen display and asynchronous AI optimization are performed in parallel.

[0006] Regarding the integration of speech recognition results with artificial intelligence optimization, existing technologies propose inputting speech recognition results into large language models or text generation models for structured transformation, style adjustment, punctuation standardization, or semantic optimization. Such solutions typically require obtaining a relatively complete recognized text first, followed by overall processing by the AI ​​model. This often results in users having to wait for the optimization results before seeing the final text, making it difficult to balance low-latency real-time display with high-quality text optimization. Furthermore, if users continue voice input or manually edit the input during optimization, existing processing methods lack mechanisms for continuous position tracking and consistency verification of the text segment being optimized, potentially leading to misplacement or overwriting of new user content.

[0007] In summary, existing technologies still lack a technical solution that can effectively coordinate the two processes of "real-time voice input triggered by mouse clicks" and "asynchronous AI optimization and accurate replacement of the input text in the background." Specifically, existing technologies have at least the following technical problems:

[0008] First, there is a latency conflict between real-time on-screen display and AI optimization. Traditional solutions either complete the recognition first and then hand it over to AI optimization for serial processing, resulting in high initial response latency and perceived lag for the user; or they only perform real-time on-screen display without AI optimization, which fails to improve the language quality of the recognition results, thus making it difficult to balance low initial response latency and high output text quality.

[0009] Secondly, in scenarios where real-time on-screen display and asynchronous AI optimization occur simultaneously, the user's voice continues to be displayed during the optimization process, and the user may also manually edit the displayed content. This will cause the text segment originally intended for optimization to drift within the displayed text. Existing "cursor coordinate positioning" solutions can only perform a one-time focus restoration after a passive interruption and cannot continuously track the position of the optimization object; while existing "overall replacement" solutions lack a position anchor mechanism, which can easily lead to problems such as misplaced replacements, overwriting new user content, or replacement in the wrong position.

[0010] Third, when the AI ​​module returns optimized candidate text, the user's current focused input control may have changed. For example, the user may have switched to another window or another input control, and the text already displayed may have been manually modified by the user. If the legality of the operation is not verified, and the replacement is directly based on the original position information, the optimization result may be incorrectly replaced with non-target controls or unexpected text positions, resulting in data corruption or a poor user experience.

[0011] Fourth, the floating window serves as a carrier for presenting the results of artificial intelligence optimization. If its display process causes the currently focused input control to lose its input focus, it will interrupt the user's ongoing voice input process, resulting in the inability to continue writing the subsequently recognized stable incremental text to the target input control, which seriously affects the continuity of real-time input.

[0012] Therefore, how to provide a mouse voice input and AI optimization method that enables uninterrupted real-time on-screen voice input, asynchronous AI optimization in the background, continuous tracking of optimized candidate text through positional anchors and accurate replacement after consistency verification, and floating window display without taking away input focus has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0013] In view of at least one of the technical problems existing in the background art, the purpose of the present invention is to provide a method, system and computer-readable storage medium for real-time mouse voice input and asynchronous optimization with artificial intelligence, so as to solve at least one of the technical problems in the traditional solution, such as mutual blocking between real-time input and artificial intelligence optimization, text segment position drift during optimization, lack of legality verification of operation, and optimization of floating window display process seizing input focus.

[0014] To achieve the above objectives, the first aspect of the present invention provides a method for real-time on-screen display of mouse voice input and asynchronous optimization using artificial intelligence, applied to an electronic device connected to a mouse. The mouse includes a smart button, a microphone, and a communication module. The electronic device runs client software capable of writing and replacing text on input controls via a system text interface. The method includes: responding to a press event of the smart button, the client software establishes a voice input session, designates the currently focused input control at the time of the press as the target input control and records the identifier of the target input control, and controls the microphone to collect user voice; performing incremental speech recognition on the user voice and differentially generating stable incremental text, writing it to the target input control in a non-blocking manner via the system text interface, and maintaining an on-screen text buffer; when unoptimized text in the on-screen text buffer from the boundary of the previous segment meets a preset character count condition, a preset pause condition, or a complete sentence condition, the unoptimized text is identified as a text segment to be optimized and a dynamic position anchor is established, the dynamic position anchor including at least the identifier of the target input control, the start offset of the text segment to be optimized, and the end offset. The process involves: verifying the text value; asynchronously sending the text segment to be optimized to the AI ​​optimization module to obtain optimization candidate text, wherein the asynchronous sending does not block the incremental speech recognition and the writing of the stable incremental text; displaying an optimization floating window for presenting the optimization candidate text in a non-active window manner that does not cause the target input control to lose input focus, wherein the optimization floating window is used for the user to perform an adoption operation or trigger a rejection event, and adjusting the offset of the dynamic position anchor point according to the writing of the stable incremental text; in response to the adoption operation, when the identifier of the target input control matches and the text verification passes, replacing the text segment to be optimized with the optimization candidate text based on the offset-corrected dynamic position anchor point; and in response to the rejection event, retaining the text segment to be optimized and closing the optimization floating window.

[0015] In the above technical solution, stable incremental text is generated differentially and written to the target input control in real time in a non-blocking manner, so that the user's voice content can be continuously presented in the input control of the user's current operation with low initial response latency. At the same time, the text segment to be optimized is sent asynchronously to the artificial intelligence optimization module, and the asynchronous sending process does not block the incremental speech recognition and the writing of the stable incremental text, so that the voice display and artificial intelligence optimization can be carried out in parallel in time, thereby avoiding the latency bottleneck of "the user must wait for the artificial intelligence optimization result" in the traditional serial processing scheme.

[0016] Furthermore, by establishing dynamic position anchors for the text segment to be optimized, including at least the identifier of the target input control, the start offset, the end offset, and the text verification value, and by continuously correcting the offset of these dynamic position anchors as stable incremental text is continuously written, the current position of the text segment to be optimized can be continuously tracked even when the user's voice is still being displayed during AI optimization. This active tracking mechanism overcomes the limitation of existing "cursor coordinate positioning" schemes, which can only perform a one-time focus restoration after passive interruption, and also overcomes the limitation of existing "overall replacement" schemes, which lack a position anchor mechanism.

[0017] Furthermore, before responding to the operation to perform the replacement, by determining that the identifier of the target input control matches and the text verification passes through this two-level verification mechanism, the optimization result can be effectively avoided from being replaced to a non-target control or an unexpected text position when the user has switched to another window or other input control, or when the text already on the screen has been manually modified by the user, thereby ensuring the legality and accuracy of the replacement operation.

[0018] In addition, by displaying the optimized floating window in a non-active window mode that does not cause the target input control to lose input focus, the display process of the optimized floating window will not interrupt the user's ongoing voice input process. The stable incremental text obtained after recognition can still be written to the target input control, thus ensuring the continuity of real-time input.

[0019] In one optional implementation, the preset character count condition is that the cumulative length of the unoptimized text reaches a preset character count threshold, which is 8 to 30 characters; the preset pause condition is that a pause in the user's speech is detected for a duration reaching a preset pause threshold, which is 300 milliseconds to 1500 milliseconds; the complete sentence condition is that the unoptimized text is determined to be a complete sentence by the sentence completeness determination module, which makes the determination based on at least one of sentence-end punctuation detection, syntactic completeness model scoring, and speech endpoint detection results. By setting the above three trigger conditions, the division of the text segment to be optimized can take into account both contextual completeness and trigger timeliness, and avoid the delay caused by optimizing an excessively long text segment all at once, thereby further balancing optimization quality and response speed.

[0020] In one optional implementation, the client software maintains the written stable recognition results. It performs a differential calculation between the portion of the current incremental recognition result obtained from the incremental speech recognition that is marked as stable or meets the preset stability criteria but has not yet been written, and the portion obtained from the differential calculation as the stable incremental text. The previous segment boundary is the position in the on-screen text buffer recorded when the text segment to be optimized was determined last time. The client software also maintains an optimization task queue, where the task states include waiting, executing, and returned states. In response to a newly triggered optimization task, when the tail optimization task in the optimization task queue is in a waiting state, and the tail optimization task corresponds to the same voice input session and the text segments are adjacent to the newly triggered optimization task, the tail optimization task is merged with the newly triggered optimization task. When there are no tail optimization tasks that can be merged, the newly triggered optimization task is added to the optimization task queue in a waiting state. When the dynamic position anchor point verification corresponding to the returned optimization candidate text fails, the optimization candidate text is marked as invalid. The differential mechanism can avoid the repeated writing or erroneous overwriting of incremental speech recognition results; the task queue and merging mechanism can reduce the number of repeated calls to the artificial intelligence optimization module and avoid repeated processing of adjacent text segments triggered multiple times in a short period of time.

[0021] In one optional implementation, the identifier of the target input control includes at least one of a window handle, a process identifier, a control identifier, and a control path. The client software determines the application type based on the identifier of the target input control and calls the corresponding optimization rule set or prompt template according to the application type. The optimization rule set or prompt template includes at least one of the following: rules for retaining colloquial filler words in instant messaging scenarios, rules for formalized expression and punctuation standardization in email scenarios, and rules for written expression and terminology dictionary matching in document editing scenarios. By calling the corresponding optimization rule set or prompt template according to different application types, the artificial intelligence optimization module can output optimization results adapted to the current usage scenario, such as retaining colloquial expressions in instant messaging scenarios, performing formalization and punctuation standardization in email scenarios, and performing written expression and terminology standardization in document editing scenarios.

[0022] In one optional implementation, the dynamic location anchor point further includes a voice input session identifier; the matching of the identifier of the target input control means comparing the identifier of the target input control in the dynamic location anchor point with the identifier of the currently focused input control when the operation occurs; the passing of text verification means comparing the verification value calculated from the current text at the corresponding position of the dynamic location anchor point with the text verification value in the dynamic location anchor point; when the client software detects that the editing position of the user editing event is before or within the range of the text segment to be optimized, it determines the appropriate method based on the user editing event. The editing position, insertion length, or deletion length of the text are used to correct the starting offset and the ending offset. The user editing events include keyboard input, input method candidate word selection, clipboard pasting, touchscreen input, or deletion operations. When the client software performs the replacement, it generates a replacement record, which includes the text before replacement, the text after replacement, and the position range of the text after replacement. In response to an undo operation on the adopted operation, when the current text within the position range of the text after replacement matches the text after replacement, the content that has been replaced in the target input control is restored to the text segment to be optimized based on the replacement record. By expanding the position anchor point into a composite structure containing the identifier of the target input control, the voice input session identifier, the starting offset, the ending offset, and the text verification value, and in conjunction with the offset correction of user editing events, the dynamic position anchor point can maintain accurate pointing to the text segment to be optimized even under multiple concurrent user operations. By performing two levels of verification before adoption and another verification of the text after replacement before undoing, it is possible to avoid erroneous restoration even if the user has manually modified the optimized content.

[0023] In one optional implementation, the rejection event is a preset event indicating that the user has not confirmed the adoption of the optimized candidate text. The rejection event includes at least one of the following: a hold operation for the optimized floating window; a timeout event for the optimized floating window; the client software detecting new content input to the target input control via a non-method text input channel during the existence of the optimized floating window, wherein the non-method text input channel includes at least one of a keyboard input channel, an input method candidate word selection channel, a clipboard paste channel, and a touch screen input channel, and the new content does not include content generated by the stable incremental text written through the system text interface; and the client software detecting that the mouse cursor moves out of a preset range of the optimized floating window, and detecting at least one of a mouse button click operation, a mouse wheel scroll operation, a keyboard key operation, and a touch operation on an area outside the optimized floating window within a preset time window after the cursor moves out. By defining the rejection event in multiple dimensions, the client software can accurately identify the user's intention not to adopt the text and avoid misjudging the stable incremental text generated by the method itself as user input and triggering a rejection.

[0024] In one optional implementation, the AI ​​optimization module includes a local optimization module and a remote optimization module. Before asynchronously sending the text segment to be optimized, the client software selects a target optimization module based on the current network status and the sensitivity level of the text segment. The sensitivity level is obtained by detecting the text segment based on keyword matching, regular expression matching, or a sensitive information identification model. When the current network status is offline or the sensitivity level reaches a preset level, the local optimization module is selected; otherwise, the remote optimization module is selected. Through the dynamic selection of local and remote optimization modules, the method of this invention can still operate in degraded mode in offline scenarios, and prioritizes local optimization to meet privacy protection requirements when sensitive information is involved.

[0025] In one optional implementation, the client software uses the release event of the smart button as the end event for microphone acquisition. After the release event occurs, it continues to perform incremental speech recognition and stable incremental text writing on the remaining unrecognized portion of the user's speech until the recognition processing of the voice input session is completed. The incremental speech recognition supports a mixed Chinese-English recognition mode and a single language recognition mode. The recognition mode is automatically selected by the client software based on the user's preset settings or the application type corresponding to the target input control. By using the release event as the end event for acquisition and continuing to process the remaining speech, the loss of speech content still in the buffer at the moment of button release can be avoided. By supporting the mixed Chinese-English recognition mode, it can adapt to the mixed Chinese-English expressions commonly found in domestic office scenarios.

[0026] A second aspect of this invention provides a system for real-time on-screen display of mouse voice input and asynchronous optimization with artificial intelligence. The system includes a mouse and an electronic device. The mouse includes smart buttons, a microphone, a main control chip, and a communication module. The smart buttons are used to receive user-triggered operations, the microphone is used to collect user voice, and the main control chip is used to send smart button status signals and voice data to the electronic device through the communication module. The electronic device includes a processor and a memory. The memory stores client software, which, when executed by the processor, is used to implement the method described in any of the first aspects of this invention.

[0027] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the first aspects of the present invention.

[0028] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0029] First, this invention processes the real-time writing of stable incremental text and the asynchronous sending of the text segment to be optimized in parallel, and explicitly limits the asynchronous sending to not block the incremental speech recognition and the writing of stable incremental text. This overcomes the latency bottleneck in the traditional "first recognize the complete text and then let artificial intelligence optimize it as a whole" scheme, where the user has to wait for the optimization result. It achieves artificial intelligence optimization while maintaining low initial response delay for real-time display of voice content, effectively balancing response speed and text quality.

[0030] Secondly, this invention establishes dynamic position anchors for the text segment to be optimized, including at least the identifier of the target input control, a start offset, an end offset, and a text verification value. As stable incremental text is continuously written and the user edits the target input control, the dynamic position anchors are offset and corrected. This ensures that even during AI optimization, when the user's voice is continuously displayed on the screen and the user is manually editing the displayed content, the current position of the text segment to be optimized can still be continuously tracked. This active tracking mechanism has a stronger position maintenance capability than the existing passive recovery scheme of "cursor coordinate positioning," which helps reduce the risk of the optimization result being replaced in the wrong position.

[0031] Third, before performing the replacement, the present invention compares whether the target input control identifier in the dynamic position anchor is consistent with the current focus input control identifier when the operation occurs, and compares whether the verification value calculated from the current text at the corresponding position of the dynamic position anchor is consistent with the text verification value in the dynamic position anchor. These two levels of verification mechanisms can effectively identify abnormal situations such as the user switching controls or modifying the original text, and will not perform the replacement if the verification fails, thereby protecting the user's current input content from being erroneously overwritten.

[0032] Fourth, the present invention uses an inactive window method to display the optimized floating window without causing the target input control to lose input focus. This ensures that the display process of the optimized floating window will not interrupt the input state of the currently focused input control. The stable incremental text obtained subsequently can still be continuously written to the target input control through the system text interface, ensuring the continuity of real-time screen display and thus avoiding the disruption of the continuity of user voice input caused by the display of the optimized floating window.

[0033] Fifth, the artificial intelligence optimization module of this invention can dynamically select between the local optimization module and the remote optimization module according to the network status and the sensitivity level of the text segment, so that the method of this invention can still run in a degraded manner in offline scenarios, and prioritize the use of local optimization when sensitive information is involved, thus taking into account both functional availability and privacy protection. At the same time, by maintaining an optimization task queue that includes three states: waiting, executing, and returned, and a merging mechanism for adjacent text segment tasks, the repeated calls to the artificial intelligence optimization module can be effectively reduced, thereby reducing system resource consumption. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the overall process of the real-time on-screen display of mouse voice input and asynchronous optimization method of artificial intelligence provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of the real-time mouse voice input display and asynchronous artificial intelligence optimization system provided in an embodiment of the present invention;

[0037] Figure 3 A timing diagram illustrating the differential generation and non-blocking writing of stable incremental text provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the data structure of the dynamic position anchor point provided in an embodiment of the present invention;

[0039] Figure 5 A schematic diagram illustrating the positional correction of the dynamic positional anchor point provided in this embodiment of the invention, based on stable incremental text writing and user editing events.

[0040] Figure 6 This is a schematic diagram of the task status and task merging mechanism of the optimized task queue provided in an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of a two-level verification process for the adoption of optimized candidate text provided in an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram illustrating the positional relationship of an optimized floating window provided in an embodiment of the present invention, which is displayed in a non-active window mode without causing the target input control to lose input focus. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0044] Referring to Figure 2, the real-time on-screen display of mouse voice input and asynchronous optimization system for artificial intelligence provided in this embodiment of the invention includes a mouse 100 and an electronic device 200. The mouse 100 and the electronic device 200 are communicatively connected, and the communication connection can be a wired connection or a wireless connection (e.g., at least one of 2.4GHz wireless, Bluetooth, and Wi-Fi).

[0045] The mouse 100 includes: a smart button 101, a microphone 102, a main control chip 103, and a communication module 104. The smart button 101 is used to receive user trigger operations, which include at least a press operation and a release operation. The microphone 102 is used to collect user voice; in this embodiment, a MEMS microphone with a digital audio output interface (e.g., PDM or I2S) can be used, and it is connected to the main control chip 103 via an audio channel. The main control chip 103 can be a general-purpose microcontroller. The communication module 104 is used to send the button status signal of the smart button 101 and the voice data collected by the microphone 102 to the electronic device 200.

[0046] The electronic device 200 may be a personal computer, laptop, tablet, or smartphone, including a processor 201 and a memory 202, wherein the memory 202 stores client software 210. The client software 210 can be functionally divided into: a voice input session management module, an incremental recognition and differential module, an on-screen text buffer, a system text interface adaptation module, a dynamic position anchor management module, an asynchronous optimization task scheduling module, an artificial intelligence optimization module, an inactive floating window management module, a two-level verification and replacement execution module, and an undo and replacement record module. The above module division is only a logical functional division; in actual implementation, the modules can be merged or further split, and those skilled in the art can select according to the specific implementation environment.

[0047] The client software 210 writes and replaces text on the target input control through a system text interface. Under the Windows operating system, the system text interface can be implemented based on at least one of the following interfaces: an input method interface (submitting stable incremental text as confirmation text to the target input control), a UI Automation interface (determining the text range, setting the selection area, or setting control values ​​through ValuePattern, TextPattern / TextPatternRange, TextEditPattern, or Selection-related interfaces), or a composite input interface (e.g., SendInput). Under macOS or Linux operating systems, it can be implemented based on an accessibility interface, a text input service interface, or the corresponding input method framework interface, respectively. This invention does not limit the specific type of the system text interface; any interface capable of writing text to the target input control and supporting replacement of a specified position range can be used as a specific implementation of the "system text interface" described in this invention.

[0048] Referring to Figure 1, the method provided in this embodiment of the invention includes steps S1 to S11. The following description uses a typical scenario: "A user is composing an email on an electronic device, and the smart button 101 is configured as a voice input button."

[0049] Step S1: The user positions the cursor in the email body editing box and presses the smart button 101. The main control chip 103 detects the press event and sends a press event signal to the electronic device 200 via the communication module 104.

[0050] Step S2: After receiving the press event signal, the client software 210 establishes a voice input session and assigns a session identifier (e.g., UUID). Simultaneously, it obtains the currently focused input control at the time of the press event through the operating system window interface, input method interface, or accessibility interface, uses it as the target input control, and records its identifier. The identifier of the target input control includes at least one of the following: window handle, process identifier, control identifier, and control path.

[0051] Steps S3 to S5: The microphone 102 continuously collects user voice and sends it to the electronic device 200 via the communication module 104; the client software 210 performs streaming speech recognition on the user voice and generates stable incremental text using the differential algorithm described below; the stable incremental text is written to the target input control in a non-blocking manner via the system text interface, and corresponding content is appended to the on-screen text buffer, so that the on-screen text buffer at least reflects the impact of the content written by this method and the detected user editing events on the conversation text. The non-blocking method can be implemented through an independent thread, an asynchronous task queue, or an asynchronous system interface.

[0052] Step S6: The client software 210 continuously monitors whether the unoptimized text in the text buffer on the screen, starting from the boundary of the previous segment, meets preset trigger conditions. The preset trigger conditions include at least one of the following: a preset character count condition (e.g., a cumulative length of 8 to 30 characters), a preset pause condition (e.g., detecting a speech pause of 300 to 1500 milliseconds), and a complete sentence condition (e.g., the unoptimized text is determined to be a complete sentence by the sentence completeness determination module). The sentence completeness determination module makes its determination based on at least one of the following: sentence-end punctuation detection, syntactic completeness model scoring, and speech endpoint detection results.

[0053] Step S7: When the preset triggering condition is met, the client software 210 identifies the unoptimized text as the text segment to be optimized and establishes a dynamic position anchor. The dynamic position anchor includes at least the identifier of the target input control, the start offset, the end offset, and the text verification value of the text segment to be optimized. It may also include the voice input session identifier and the base offset (used to map to the full-text position of the target input control). The text verification value can be generated using hash algorithms with low collision probability and low computational cost, such as CRC32, Adler32, or SHA-1 digest truncation value. After establishing the dynamic position anchor, the boundary of the previous segment is updated to the end offset of the text segment to be optimized in this iteration.

[0054] Step S8: The client software 210 asynchronously sends the text segment to be optimized and the current application context information to the artificial intelligence optimization module to obtain optimization candidate text. This asynchronous sending is executed using an independent thread or asynchronous task, without blocking the incremental speech recognition and subsequent stable incremental text writing. In other words, while the optimization task waits for the return result, the microphone 102 continues to collect speech, and the client software 210 continues to generate new stable incremental text and write it to the target input control.

[0055] Step S9: After the AI ​​optimization module returns the optimization candidate text, the client software 210 displays an optimization floating window in a deactivated window mode that does not cause the target input control to lose input focus. The optimization floating window is used to present the optimization candidate text and two types of user operation controls: "Adopt" and "Reject". During the display of the optimization floating window, the client software 210 performs offset correction on the dynamic position anchor point based on the subsequently written stable incremental text and detected user editing events.

[0056] Step S10: The user performs an operation on the optimized floating window:

[0057] (a) If the user performs an adoption operation (e.g., clicks the "Adopt" control or presses a preset shortcut key), the client software 210 performs two levels of verification: First, it compares the identifier of the target input control in the dynamic position anchor with the identifier of the currently focused input control when the adoption operation occurs; Second, it calculates the verification value of the current text at the corresponding position of the dynamic position anchor and compares it with the text verification value stored in the dynamic position anchor. After both levels of verification pass, based on the offset-corrected dynamic position anchor, the text segment to be optimized is replaced with the optimization candidate text via the system text interface, and a replacement record containing the text before replacement, the text after replacement, and the position range of the text after replacement is generated for later cancellation; if the verification fails, the replacement is not performed.

[0058] (b) If a rejection event occurs, the text segment to be optimized is retained unchanged and the optimization floating window is closed. The rejection event includes at least one of the following: a retention operation for the optimization floating window, a timeout event for the optimization floating window, the client software 210 detecting new content input to the target input control via a text input channel other than that of this method (including at least one of keyboard, input method candidate word selection, clipboard pasting, or touch screen input channel, excluding the stable incremental text writing of this method itself) during the existence of the optimization floating window, and at least one of the following: the mouse cursor moves out of the preset range of the optimization floating window and at least one of the following is detected within a preset time window: mouse button click, scroll wheel scrolling, keyboard key press, or touch operation outside the floating window.

[0059] Step S11: The user releases the smart button 101. The client software 210 uses the release event as the end event for microphone acquisition, but continues to perform incremental speech recognition and stable incremental text writing on the remaining parts of the user's voice that have not yet been recognized, until the recognition processing of the voice input session is completed. After the recognition processing of the voice input session is completed and all the optimization tasks triggered by it have been adopted, rejected, or marked as invalid, the voice input session is destroyed.

[0060] Referring to Figure 3, the client software 210 maintains a stable recognition result sequence S, initially set to an empty string. The streaming speech recognition engine periodically outputs the current complete recognition hypothesis H and a stability marker. The stability marker can be the length L of the stabilized prefix; if the engine does not directly provide L, it can be inferred from the longest common prefix length of the current recognition hypothesis H and the previous recognition hypothesis H_prev. Whenever the streaming speech recognition engine outputs a new recognition hypothesis, the client software 210 executes the following difference algorithm: extracts the first L characters from H as the current stable prefix H_stable; if H_stable uses S as a prefix, the remaining part of H_stable after removing the prefix S is used as the newly added stable incremental text Δ; if Δ is not empty, S is updated to H_stable and Δ is returned; if H_stable does not use S as a prefix, it is determined that the current stable prefix is ​​inconsistent with the written stable recognition result, no new writing is performed, and the current speech input session is marked as needing resynchronization. When the system text interface supports deletion and replacement operations, the client software 210 can also calculate the longest common prefix C of S and H_stable, delete the content in the target input control that corresponds to the tail content of S after C, write the remaining part of H_stable after C as the new stable incremental text, and update S to H_stable.

[0061] It should be understood that the above differential algorithm is merely an example. Any method capable of extracting a "stable sequence of characters newly added since the last write" from the continuous output of a streaming speech recognition engine can be used as a specific implementation of the "differential generation of stable incremental text" described in this invention.

[0062] Referring to Figures 4 and 5, the dynamic position anchor object is represented in a structured data format, including at least the following fields: anchor unique identifier (anchorId), associated voice input session identifier (sessionId), target input control identifier (controlId, including at least one of window handle, process identifier, control path, and control automation identifier), base insertion position (baseOffset, the insertion starting point of the current voice input session in the target input control), start offset (startOffset, relative to the on-screen text buffer), end offset (endOffset), text checksum, and anchor state (state, valid or invalid). The client software 210 uses baseOffset, startOffset, and endOffset to jointly determine the actual replacement range in the target input control.

[0063] For example, in one specific implementation, the dynamic position anchor point can be represented as the following structured data: { "anchorId": "A001", "sessionId": "S001", "controlId": { "windowHandle": "0x001A03F2", "processId": "3568", "controlPath": " / Window / Edit[1]", "automationId": "mailBody" }, "baseOffset": 120, "startOffset": 8, "endOffset": 18, "textChecksum": "9F2A31C8", "state": "VALID" }

[0064] In this dataset, `anchorId` represents the unique identifier of the anchor point, `sessionId` represents the identifier of the voice input session, `controlId` represents the identifier of the target input control, `baseOffset` represents the baseline insertion position of the current voice input session within the target input control, `startOffset` and `endOffset` represent the starting and ending offsets of the text segment to be optimized within the text buffer on the screen, respectively, `textChecksum` represents the text checksum of the text segment to be optimized, and `state` represents the anchor point state. The above structured data is merely an example; in actual implementations, it can be represented using any of the following formats: JSON, key-value objects, structs, database records, or in-memory objects.

[0065] After the text segment to be optimized is asynchronously sent, the dynamic position anchor management module continuously monitors stable incremental text writing events and user editing events, and performs offset corrections on the baseOffset, startOffset, and endOffset of the dynamic position anchor. The user editing events include at least one of keyboard input, input method candidate word selection, clipboard pasting, touchscreen input, deletion operation, or selection replacement. Each event carries the editing position editPos, insertion length insLen, and deletion length delLen. If an editing event is detected in the target input control before the baseOffset, the baseOffset is adjusted accordingly. For editing events relative to the text buffer on the screen, the offset correction rules are as follows:

[0066] Rule 1: If editPos + delLen ≤ startOffset, or delLen = 0 and editPos ≤ startOffset, then startOffset_new = startOffset + insLen - delLen, endOffset_new = endOffset + insLen - delLen;

[0067] Rule 2: If the scope of an edit event overlaps with (startOffset, endOffset), or if the editPos of a pure insertion event is within (startOffset, endOffset), then the edit event is determined to affect the scope of the text segment to be optimized. When the deletion or replacement operation completely covers the text segment to be optimized, the dynamic position anchor point is marked as invalid; otherwise, the endOffset is corrected according to the insertion length and deletion length, and the textChecksum is recalculated, or the text validation is considered to have failed when the operation is performed.

[0068] Rule 3: If editPos ≥ endOffset and the affected range does not overlap with (startOffset, endOffset), the dynamic position anchor is not affected.

[0069] Illustrated with a specific scenario: At time t0, the content in the on-screen text buffer is "The weather is great today. Let's go to the park together." (14 characters), and the text segment to be optimized is "The weather is great" (start = 2, end = 6); at time t1, new text "看花" is written onto the screen by voice (appended to the end of the buffer, and the anchor position remains unchanged); at time t2, the user inserts the character "好" at position 2 in the buffer (delLen = 0, editPos = 2, applicable to the path of "pure insertion event and editPos ≤ startOffset" in Rule 1), and after correction, start = 3, end = 7. If the user performs subsequent operations, based on the corrected anchor position, the text segment to be optimized "The weather is great" can be accurately replaced with the optimization candidate text.

[0070] Referring to FIG. 6, the client software 210 maintains an optimization task queue, and the task objects in the queue at least include: task unique identifier, session identifier to which it belongs, associated anchor identifier, content of the text segment to be optimized, current application context information, task status, and optimization candidate text field. The task status includes waiting status (WAITING), executing status (EXECUTING), returned status (RETURNED), merged status (MERGED), and invalid status (INVALID). The working thread or asynchronous execution slot only fetches tasks with a status of WAITING for sending, and skips tasks with a status of MERGED or INVALID.

[0071] When a newly triggered optimization task enters, the client software 210 processes it according to the following logic: It retrieves the tail task of the queue; if the queue is empty, it directly enqueues the newly triggered optimization task in the WAITING state; if the tail task is in the WAITING state, its session identifier is the same as the newly triggered optimization task, and its associated anchor's endOffset is equal to the newly triggered optimization task's associated anchor's startOffset, then the tail task and the newly triggered optimization task are merged into a merged task. The merged task's inputText is the concatenation of two text segments, the anchor is the merged range, and the textChecksum is recalculated; the original tail task is marked as MERGED and no longer executed; the merged task is added to the queue in the WAITING state, and the newly triggered optimization task is no longer enqueued as an independent task; otherwise, the newly triggered optimization task is enqueued in the WAITING state. This task merging mechanism reduces repeated calls to the AI ​​optimization module and enables the AI ​​optimization module to provide higher-quality optimization results based on a more complete context.

[0072] The AI ​​optimization module includes a local optimization submodule and a remote optimization submodule. Before asynchronously sending the text segment to be optimized, the client software 210 selects a target optimization submodule based on the current network status and the sensitivity level of the text segment to be optimized: when the current network status is offline or the sensitivity level reaches a preset level, the local optimization submodule is selected; otherwise, the remote optimization submodule is selected. The sensitivity level is obtained after detecting the text segment to be optimized based on keyword matching, regular expression matching, or a sensitive information recognition model.

[0073] The client software 210 further determines the application type based on the identifier of the target input control, and calls the corresponding optimization rule set or prompt template according to the application type. For example, in an instant messaging scenario, rules that retain colloquial filler words are used; in an email scenario, formal expressions and punctuation rules are used; and in a document editing scenario, written expressions and terminology dictionary matching rules are used. In a remote optimization scenario, the prompt template can be input to the large language model in the form of system prompt words; in a local optimization scenario, the optimization rule set can be implemented in the form of a rule engine or a local small model.

[0074] Referring to Figure 7, the two-level verification process is triggered when the user performs an adoption operation: First, an identifier comparison is performed—the identifier of the target input control in the dynamic position anchor is compared with the identifier of the currently focused input control at the time the adoption operation occurs; if the identifiers do not match (e.g., the user has switched to another window or another input control), no replacement is performed. If the identifiers match, text verification is performed—the verification value of the current text at the corresponding position of the dynamic position anchor (mapped to the target input control according to baseOffset, start offset, and end offset) is calculated and compared with the textChecksum stored in the dynamic position anchor; if the verification values ​​do not match (e.g., the text already displayed has been manually modified by the user), no replacement is performed. After both levels of verification pass, the text segment to be optimized is replaced with the optimization candidate text via the system text interface.

[0075] Referring to Figure 8, after the artificial intelligence optimization module returns the optimized candidate text, the inactive floating window management module displays the optimized floating window in an inactive window mode that does not cause the target input control to lose input focus. On the Windows platform, this inactive window mode can be achieved by setting the window extension style WS_EX_NOACTIVATE or an equivalent attribute; on the macOS platform, it can be achieved using NSPanel in conjunction with attributes such as nonactivatingPanel; on other platforms, similar mechanisms can be used to prevent the floating window from seizing focus. During the display of the optimized floating window, the target input control can still receive stable incremental text written by the client software 210 through the system text interface. The operation can be triggered by inactive window mouse events, global shortcut keys, or mouse messages captured by the client software.

[0076] When performing a replacement, the client software 210 generates a replacement record, which includes at least the text before replacement, the text after replacement, and the position range of the text after replacement. In response to an undo operation for the adopted operation, the client software 210 first verifies whether the current text within the position range of the text after replacement matches the text after replacement in the replacement record (to prevent accidental restoration if the user has manually modified the optimized content). If they match, the software restores the replaced content in the target input control to the text segment to be optimized based on the replacement record.

[0077] Furthermore, after the release event occurs, the client software 210 continues to perform incremental speech recognition and stable incremental text writing on the remaining unrecognized parts of the user's voice until the recognition processing of the voice input session is completed, in order to avoid the loss of voice content still in the buffer at the moment of key release. The incremental speech recognition supports mixed Chinese and English recognition mode and single language recognition mode. The recognition mode can be automatically selected by the client software 210 according to the user's preset settings or the application type corresponding to the target input control.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time on-screen display of mouse voice input and asynchronous optimization with artificial intelligence, applied to an electronic device communicating with a mouse, wherein the mouse includes smart buttons, a microphone, and a communication module, and the electronic device runs client software capable of writing and replacing text on input controls via a system text interface, characterized in that... The method includes: In response to the press event of the smart button, the client software establishes a voice input session, uses the currently focused input control at the time of the press as the target input control and records its identifier, and controls the microphone to collect the user's voice. Incremental speech recognition is performed on the user's speech, and stable incremental text is generated by differential processing. The text is written to the target input control in a non-blocking manner through the system text interface, and a text buffer for the screen is maintained. When the unoptimized text in the text buffer above the screen, starting from the boundary of the previous segment, meets the preset word count condition, preset pause condition, or complete sentence condition, the unoptimized text is identified as a text segment to be optimized and a dynamic position anchor is established. The dynamic position anchor includes at least the identifier of the target input control, the start offset of the text segment to be optimized, the end offset, and the text verification value. The text segment to be optimized is asynchronously sent to the artificial intelligence optimization module to obtain optimization candidate text. The asynchronous sending does not block the incremental speech recognition and the writing of the stable incremental text. An optimization floating window for presenting the optimized candidate text is displayed in an inactive window manner that does not cause the target input control to lose input focus. The optimization floating window is used for the user to perform an adoption operation or trigger a rejection event, and the dynamic position anchor point is offset and corrected according to the writing of the stable incremental text. In response to the adoption operation, when the identifier of the target input control is consistent and the text validation passes, the text segment to be optimized is replaced with the optimization candidate text based on the offset-corrected dynamic position anchor point; in response to the rejection event, the text segment to be optimized is retained and the optimization floating window is closed.

2. The method according to claim 1, characterized in that, The preset character count condition is that the cumulative length of the unoptimized text reaches a preset character count threshold, which is 8 to 30 characters; the preset pause condition is that the detected pause in the user's speech reaches a preset pause threshold, which is 300 milliseconds to 1500 milliseconds; the complete sentence condition is that the unoptimized text is determined to be a complete sentence by the sentence completeness determination module, which makes the determination based on at least one of sentence-end punctuation detection, syntactic completeness model scoring, and speech endpoint detection results.

3. The method according to claim 1, characterized in that, The differential generation of stable incremental text includes: the client software maintaining written stable recognition results, performing differential analysis on the portion of the current incremental recognition result obtained from the incremental speech recognition that is marked as stable or meets the preset stability judgment condition but has not yet been written, and using the newly added stable portion obtained by differential analysis as the stable incremental text; the previous segment boundary is the position in the on-screen text buffer recorded when the text segment to be optimized was determined last time; the client software also maintains an optimization task queue, the task status in the optimization task queue includes waiting status, execution status, and returned status; in response to a newly triggered optimization task, when the tail optimization task in the optimization task queue is in a waiting state, and the tail optimization task and the newly triggered optimization task correspond to the same voice input session and the text segments are adjacent, the tail optimization task and the newly triggered optimization task are merged; when there is no tail optimization task that can be merged, the newly triggered optimization task is added to the optimization task queue in a waiting state; when the dynamic position anchor point verification corresponding to the returned optimization candidate text fails, the optimization candidate text is marked as invalid.

4. The method according to claim 1, characterized in that, The identifier of the target input control includes at least one of window handle, process identifier, control identifier, and control path; the client software determines the application type based on the identifier of the target input control, and calls the corresponding optimization rule set or prompt template according to the application type. The optimization rule set or prompt template includes at least one of the following: rules for retaining colloquial filler words in instant messaging scenarios, rules for formal expression and punctuation standardization in email scenarios, rules for written expression and terminology dictionary matching in document editing scenarios.

5. The method according to claim 1, characterized in that, The dynamic position anchor also includes a voice input session identifier; identifier consistency means comparing the identifier of the target input control in the dynamic position anchor with the identifier of the currently focused input control when the adoption operation occurs; text verification passing means comparing the verification value calculated from the current text at the corresponding position of the dynamic position anchor with the text verification value in the dynamic position anchor; when the client software detects that the editing position of the user editing event is before or within the range of the text segment to be optimized, it corrects the starting offset and the ending offset according to the editing position, insertion length, or deletion length of the user editing event, wherein the user editing event includes keyboard input, input method candidate word selection, clipboard pasting, touch screen input, or deletion operation; when the client software performs the replacement, it generates a replacement record, which includes the text before replacement, the text after replacement, and the position range of the text after replacement; in response to the undo operation for the adoption operation, when the current text within the position range of the text after replacement is consistent with the text after replacement, the content that has been replaced in the target input control is restored to the text segment to be optimized based on the replacement record.

6. The method according to claim 1, characterized in that, The rejection event is a preset event indicating that the user has not confirmed the adoption of the optimized candidate text, and the rejection event includes at least one of the following: (a) The retention operation for the optimized floating window; (b) The timeout event of the optimized floating window; (c) During the existence of the optimized floating window, the client software detects new content input to the target input control via a non-method text input channel, wherein the non-method text input channel includes at least one of a keyboard input channel, an input method candidate word selection channel, a clipboard paste channel, and a touch screen input channel, and the new content does not include the content generated by the stable incremental text written through the system text interface; (d) The client software detects that the mouse cursor moves out of the preset range of the optimized floating window, and detects at least one of the following within a preset time window after the cursor moves out: mouse button click operation, mouse wheel scroll operation, keyboard key operation, and touch operation on the area outside the optimized floating window.

7. The method according to claim 1, characterized in that, The artificial intelligence optimization module includes a local optimization module and a remote optimization module. Before asynchronously sending the text segment to be optimized, the client software selects the target optimization module based on the current network status and the sensitivity level of the text segment to be optimized. The sensitivity level is obtained after detecting the text segment to be optimized based on keyword matching, regular expression matching, or a sensitive information recognition model. When the current network status is offline or the sensitivity level reaches a preset level, the local optimization module is selected; otherwise, the remote optimization module is selected.

8. The method according to claim 1, characterized in that, The client software uses the release event of the smart button as the end event of the microphone acquisition, and continues to perform incremental speech recognition and stable incremental text writing on the remaining part of the user's voice that has not yet been recognized after the release event occurs, until the recognition processing of the voice input session is completed; the incremental speech recognition supports mixed Chinese and English recognition mode and single language recognition mode, and the recognition mode is automatically selected by the client software according to the user's preset or the application type corresponding to the target input control.

9. A system for real-time on-screen display of mouse voice input and asynchronous optimization with artificial intelligence, characterized in that, The system includes a mouse and an electronic device; the mouse includes a smart button, a microphone, a main control chip, and a communication module, the smart button is used to receive user-triggered operations, the microphone is used to collect user voice, and the main control chip is used to send smart button status signals and voice data to the electronic device through the communication module; the electronic device includes a processor and a memory, the memory stores client software, and the client software, when executed by the processor, is used to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images