A multi-modal data delivery method based on mouse key triggering

Abstract

This application discloses a multimodal data delivery method based on mouse button triggering. The method includes: obtaining an access description record of the target third-party intelligent agent, including candidate delivery paths and the data types and adaptation rules supported by each path; acquiring audio and selecting screen objects during the period from button press to release to obtain instruction text and object data; encapsulating the instruction text, object data, and button identifier into a unified format interactive data packet, and labeling the object data type; determining the target delivery path based on the labeled data type, the data types supported by each path, and their availability status, and delivering the object data corresponding to data types not supported by the path after conversion; the delivery confirmation result is divided into received, not received, and unconfirmed; if not received and other available paths exist, re-delivery is performed through those available paths; if unconfirmed, re-delivery is not automatically performed. This method enables multimodal data delivery for third-party intelligent agents with different access capabilities.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a multimodal data delivery method, electronic device and computer-readable storage medium based on mouse button triggering. Background Technology

[0002] With the development of artificial intelligence technology, the number of third-party intelligent agent clients running on electronic devices is increasing. Users often need to associate objects on the screen with their own intentions while using electronic devices. Figure 1 The data is then handed over to a third-party intelligent agent for processing. The object could be a selected piece of text, a file, or an area on the screen. Related technologies have already developed methods that use mouse buttons to trigger voice capture and combine this with cursor movements on the screen to select objects. The user presses a button, speaks their intention, and selects the object; after releasing the button, the electronic device sends the data to an artificial intelligence processing module.

[0003] However, when the content delivered to the recipients triggered by mouse clicks is extended from a specific artificial intelligence processing module to multiple third-party intelligent agent clients developed by different manufacturers, the above method exhibits the following problems in actual use.

[0004] First, the access capabilities offered by different third-party AI agents vary considerably. Some provide APIs, others include a software development kit (SDK) with their installation package, and still others do not offer these access methods, only providing a graphical interface for user interaction. For the latter, users need to manually navigate to the AI ​​agent's window, locate the input field, and manually enter and submit the content. When the same content is sent to different third-party AI agents, the user's actions differ, thus disrupting the interaction process.

[0005] Secondly, different access methods support different data types. When sending data via API calls or software development kits, multiple data types, including text, files, and images, are typically supported. However, in some graphical interfaces that only provide text input controls, files and images cannot be directly sent via these controls. This means that objects selected in the same operation may be delivered completely to some third-party agents, but not in their original form to others, thus limiting the use cases involving files and screen areas.

[0006] Third, after content is sent, users have difficulty knowing whether it has been received by the third-party agent. Third-party agents respond to sent content in various ways: some provide a response, some show changes on their interface, and some do not respond if they are not running or their window is not active. Users need to switch their attention to the third-party agent's interface for confirmation; if they send the content again before confirmation, the same content may be executed twice by the third-party agent; if they do not send it again, they may be in a waiting state for a long time.

[0007] Therefore, the relevant technologies have the following problems: When dealing with multiple third-party intelligent agents with inconsistent access capabilities, it is difficult to deliver multimodal content obtained from the same mouse button trigger in a unified manner. In particular, content involving files and screen areas cannot be delivered in its original form on third-party intelligent agents that can only input text, and it is difficult to know whether the content has been received after it has been sent. Summary of the Invention

[0008] This application provides a multimodal data delivery method, electronic device, and computer-readable storage medium based on mouse button triggering, to solve the problem that when multiple third-party intelligent agents with inconsistent access capabilities are involved, the multimodal content obtained from the same mouse button trigger is difficult to deliver in a unified manner. In particular, content involving files and screen areas cannot be delivered in its original form on third-party intelligent agents that can only input text, and it is difficult to know whether the content has been received after it has been sent.

[0009] In a first aspect, this application provides a multimodal data delivery method based on mouse button triggering, applied to an electronic device communicatively connected to a mouse, wherein the mouse includes smart buttons and a microphone, and the electronic device runs client software, the method comprising:

[0010] S1, obtain the access description record of the target third-party intelligent agent, the access description record includes the candidate delivery path of the third-party intelligent agent and the data types supported by each candidate delivery path and the corresponding adaptation rules;

[0011] S2, responding to the press event of the smart button to start audio acquisition and screen object selection, responding to the release event of the smart button to terminate the audio acquisition and screen object selection, and obtaining the instruction text obtained by speech recognition of the acquired audio and the selected object data.

[0012] S3, encapsulate the instruction text and the object data into an interactive data packet of a unified format, and label the data type to which the object data belongs;

[0013] S4, obtain the availability status of each candidate delivery path, determine the target delivery path based on the marked data type, the data types supported by each candidate delivery path and the availability status; according to the adaptation rules, convert the object data in the interaction data packet corresponding to the data type not supported by the target delivery path into the data type supported by the target delivery path, and then convert the converted interaction data packet into the delivery form corresponding to the target delivery path before delivery;

[0014] S5. Obtain a delivery confirmation result based on the target delivery path. The delivery confirmation result is either received, not received, or cannot be confirmed. If the delivery confirmation result is not received and there are other available candidate delivery paths, re-deliver through the other available candidate delivery paths. If the delivery confirmation result is cannot be confirmed, do not automatically re-deliver.

[0015] In some implementations, the candidate delivery path includes at least two of the following: interface call path, component call path, and interface writing path. The access description record corresponding to the interface writing path also includes the target process. The data types supported by the interface call path and the component call path include text, file, and image types. The data type supported by the interface writing path includes text. The target third-party intelligent agent is a third-party intelligent agent pre-set in the client software, or a third-party intelligent agent determined based on the user's selection in the client software. The access description record is established through at least one of the following methods: active registration by the third-party intelligent agent, writing during software development kit installation, pre-setting in the client software, user configuration, and automatic discovery based on applications installed on the electronic device. Determining the target delivery path includes: excluding candidate delivery paths whose available status is unavailable, prioritizing candidate delivery paths whose supported data types cover the marked data type over candidate delivery paths whose supported data types do not cover the marked data type, sorting the candidate delivery paths in the same order according to a preset path priority, and selecting the one ranked first as the target delivery path.

[0016] In some implementations, obtaining the availability status of each of the candidate delivery paths includes: the availability status of the interface call path is determined based on the connectivity detection result of the interface corresponding to the path; the availability status of the component call path is determined based on the loading result of the software development kit corresponding to the path; the availability status of the interface writing path is determined based on whether the target process is in a running state. If the target process is not in a running state, the target process is started and the availability status is determined to be available after the target window to which the target process belongs appears.

[0017] In some implementations, the object data includes at least one of selected text, file identifier, cursor trajectory data, and screen area; the interaction data package also encapsulates the key identifier of the smart button; the screen object selection includes: recording the position sequence of the mouse cursor on the screen during the smart button press; determining the selection range based on the start and end points of the position sequence; when the area of ​​the selection range is less than a preset area threshold, the selectable element at the start point position is used as the selection result; otherwise, the content within the selection range is used as the selection result; based on the type of the host application to which the selection result belongs and the content format of the selection result, the selection result is determined to be one of the selected text, the file identifier, or the screen area, and used as the object data; the cursor trajectory data includes the position sequence.

[0018] In some implementations, converting object data in the interactive data packet corresponding to a data type not supported by the target delivery path into a data type supported by the target delivery path includes: when the object data is a file identifier and the target delivery path does not support the file type, converting the file identifier into the path text of the file on the electronic device, or into text extracted from the content of the file; when the object data is a screen area and the target delivery path does not support the image type, using the resource location identifier obtained after uploading the area image corresponding to the screen area, or the text obtained by performing text recognition on the area image, as the conversion result; and recording the original data type, the converted data type, and the conversion method identifier involved in this conversion in the converted interactive data packet.

[0019] In some implementations, when the target delivery path is the interface writing path, the delivery includes: activating the target window to which the target process belongs; identifying the control with input focus and editable attributes in the target window as the target input control; if the control does not exist, traversing the control tree of the target window and scoring each candidate input control based on at least two of the control's type identifier, size, relative position, and prompt text; selecting the highest-scoring control as the target input control and setting its input focus; verifying that the control currently with input focus is consistent with the target input control and that the process to which the target window belongs is consistent with the target process before writing; writing and performing a submission operation after passing the verification; before identifying the control with input focus and editable attributes in the target window as the target input control, it also includes determining whether the control is in an input-enabled state based on whether there is a text insertion character in the control; only when it is in an input-enabled state is it identified as the target input control; the scoring of each candidate input control includes: the control's class The score of a candidate input control is increased when the type identifier belongs to a preset input control type, when the control size is greater than a preset size threshold, when the relative position of the control in the target window is within a preset area, and when the prompt text of the control matches a preset prompt word set. Multiple interactive data packets corresponding to the same target process are delivered sequentially in a serial queue. The next interactive data packet is delivered only after the delivery confirmation result of the previous interactive data packet is determined to be received or not received. Obtaining the delivery confirmation result based on the target delivery path includes: observing the target input control at a preset sampling period after the submission operation; using the change of the content of the target input control from the written content to emptiness as the criterion for acceptance; using the appearance of a new error prompt control in the target window as the criterion for non-acceptance; and determining that the content of the target input control is still consistent with the written content within a preset time period, indicating that it cannot be confirmed.

[0020] In some implementations, obtaining the delivery confirmation result based on the target delivery path includes: when the target delivery path is the interface call path, using the presence of a receiving identifier or task identifier in the received response message as the criterion for determining whether the delivery confirmation result has been received, and using the presence of a rejection identifier or call failure identifier in the received response message as the criterion for determining whether the delivery confirmation result has not been received, and determining that the delivery confirmation cannot be made if no response message is received within a preset time period; when the target delivery path is the component call path, using the callback event returned by the software development kit belonging to a preset receiving event type as the criterion for determining whether the delivery confirmation result has been received, and using the callback event belonging to a preset exception event type as the criterion for determining whether the delivery confirmation result has not been received, and determining that the delivery confirmation cannot be made if no callback event is received within a preset time period.

[0021] In some implementations, the client software assigns a delivery identifier to the interactive data packet, and carries the delivery identifier with each delivery; when re-delivering, the delivery identifier is reused, and the candidate delivery path used for this re-delivery is recorded in the delivery record corresponding to the delivery identifier; when the delivery confirmation result is that it cannot be confirmed, the client software keeps the interactive data packet in a pending state and outputs a prompt, and only re-delivers it after receiving a confirmation operation from the user via the mouse.

[0022] In some embodiments, the mouse further includes a light output unit, and the method further includes: determining a light effect state based on the delivery confirmation result, and sending the determined light effect state to the light output unit, wherein the light effect states corresponding to the delivery confirmation result being received, not received, and unable to be confirmed are different from each other; when the target delivery path is the interface writing path and is in the observation period of the target input control, the light output unit outputs the light effect state corresponding to the observation.

[0023] In a second aspect, this application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the method described in the first aspect above.

[0024] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0025] In the method provided in this application, by obtaining the access description record of the target third-party intelligent agent, the same interactive data packet can determine the delivery path to be used based on its labeled data type, the data types supported by each candidate delivery path, and the availability status of each candidate delivery path; for data types not supported by the determined delivery path, the corresponding object data is converted and sent out together with the interactive data packet; for third-party intelligent agents that do not provide calling interfaces and software development kits, they are written and submitted through the input controls in their graphical interface, and the input focus and the process to which they belong are verified before writing to determine the writing position; the delivery confirmation result is divided into three types: received, not received, and unable to confirm. If it is not received and there are other available candidate delivery paths, it is re-delivered through other available candidate delivery paths; if it is unable to confirm, no automatic re-delivery occurs, and the interactive data packet remains in a pending state and waits for the user's confirmation operation via the mouse input. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the application environment of the multimodal data delivery method based on mouse button triggering provided in the embodiments of this application.

[0027] Figure 2 A flowchart illustrating the multimodal data delivery method based on mouse button triggering provided in this application embodiment;

[0028] Figure 3 This is a schematic diagram illustrating the process of determining the target delivery path and performing data type conversion in an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the data delivery process via writing a path through the interface in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the state transition of the delivery confirmation result and the corresponding lighting effect state in the embodiments of this application. Detailed Implementation

[0031] The technical solutions in the embodiments of this application are described below with reference to the accompanying drawings. The described embodiments are only some embodiments of this application, and not all embodiments.

[0032] I. Application Environment

[0033] See Figure 1 The application environment of this application embodiment includes a mouse, an electronic device, and multiple third-party intelligent agent clients running on the electronic device. The mouse is communicatively connected to the electronic device, and the communication connection can be wired or wireless. The mouse includes smart buttons and a microphone, and in some embodiments, it also includes a light output unit. The smart buttons are buttons separately provided on the mouse other than the left button, right button, and scroll wheel, or they can be buttons that reuse one of the above buttons and are distinguished by firmware.

[0034] The electronic device runs client software that establishes a data channel with the mouse, receives press and release events from the smart button and audio collected by the microphone, and sends lighting effect status to the lighting output unit. The third-party smart agent clients are developed by different manufacturers, and their external access capabilities differ: some provide API calls, some include a software development kit (SDK) with their installation package, and some only provide a graphical interface. Correspondingly, there are API call paths, component call paths, and interface writing paths between the client software and each of the third-party smart agent clients.

[0035] II. Main Method Flow

[0036] See Figure 2 The multimodal data delivery method based on mouse button triggering provided in this application includes steps S1 to S5.

[0037] S1 obtains access description record

[0038] The client software stores access description records for each third-party intelligent agent. These access description records, using a key-value structure, relational structure, or other queryable data structure, are stored in the electronic device's local configuration file, local database, or other readable and writable storage location. Their content includes: third-party intelligent agent identifiers, a list of candidate delivery paths, the data types supported by each candidate delivery path, and the adaptation rules corresponding to each candidate delivery path. The access description record corresponding to the interface write path also includes the target process, which is represented by an executable file name, process name, or application package identifier.

[0039] An example of an access description record is as follows: the third-party intelligent agent is identified as A; the candidate delivery path includes the interface call path and the interface writing path; the interface call path supports text type, file type and image type, and its adaptation rules record the request address, request method, authentication method and the field positions of instruction text, object data and data type in the request body; the interface writing path supports text type, and its adaptation rules record the target process, window title characteristics, input control type, writing method and submission method.

[0040] The access description record is established through at least one of the following methods: a third-party intelligent agent actively registers with the client software when it is installed or started; it is written by the installer when it is installed with the software development kit; it is preset by the client software; it is filled in by the user in the configuration interface of the client software; or it is automatically discovered by the client software based on the list of applications installed on the electronic device and generated according to a preset template.

[0041] The target third-party intelligent agent is a pre-defined third-party intelligent agent in the client software. In other embodiments, the client software interface lists each third-party intelligent agent with established access description records, and the target third-party intelligent agent is determined based on the user's selection therein. In still other embodiments, the electronic device is connected to multiple mice, or a single mouse is equipped with multiple smart buttons, with different button identifiers corresponding to different third-party intelligent agents, and the target third-party intelligent agent is determined based on the button identifier triggered this time.

[0042] S2 Acquisition Command Text and Object Data

[0043] The client software listens for press events of the smart button. In response to the press event, the client software initiates audio acquisition and screen object selection; in response to the release event of the smart button, it terminates audio acquisition and screen object selection. The time interval between the press event and the release event constitutes one trigger.

[0044] The collected audio is used to perform speech recognition to obtain the instruction text. The speech recognition is performed locally on the electronic device, or the client software can send the collected audio to a speech recognition service and retrieve the recognition result.

[0045] The selection of screen objects is performed as follows: the client software records the position of the mouse cursor on the screen at preset sampling intervals during the smart button press, obtaining a position sequence; the selection range of a rectangle is determined using the start and end points of the position sequence as diagonal vertices. When the area of ​​the selection range is less than a preset area threshold, the operation is determined to be a click, and the selectable element at the start point is used as the selection result; when the area of ​​the selection range is not less than the preset area threshold, the operation is determined to be a box selection, and the content within the selection range is used as the selection result. The preset area threshold is set according to the screen resolution, mouse sensitivity, or user configuration, for example, set to 100 square pixels, that is, when the cursor movement during the press does not exceed 10 pixels horizontally or vertically, it is treated as a click.

[0046] Based on the type of the host application to which the selection result belongs and the content format of the selection result, the selection result is determined as one of selected text, file identifier, or screen area, and used as object data: when the host application is a file manager and the selection result is a file item within it, the complete path of the file is taken as the file identifier; when the host application is a text editor, web browser, or document reader and the selection result contains extractable characters, the extracted characters are taken as the selected text; when neither of the above is true, the coordinates and size of the selection area on the screen are taken as the screen area. The object data obtained in a single trigger can be one of the above three, or two or more of them, for example, multiple file identifiers are obtained when multiple file items are selected in a file manager.

[0047] The position sequence itself can also serve as the object data, referred to as cursor trajectory data, which consists of a series of cursor positions recorded during the pressing of the smart button. When the object data is the cursor trajectory data, whether there is a selectable element within the selection range does not affect the acquisition of the cursor trajectory data. The cursor trajectory data is the movement trajectory of the cursor drawn on the screen in the shape of a hook, circle, or other shapes. In other embodiments, the cursor trajectory data and at least one of the selected text, the file identifier, or the screen area are obtained simultaneously in a single trigger.

[0048] In some embodiments, when the release event occurs, there is only one position in the position sequence, and the selectable element at that position is used as the selection result. In still other embodiments, if no audio is collected or the speech recognition result is empty during the smart button press, the instruction text is an empty string, and subsequent steps are executed as usual.

[0049] S3 encapsulates interactive data packets.

[0050] The instruction text, the object data, and the key identifier of the smart button are encapsulated into a unified format interaction data packet, and the data type of the object data is marked in the interaction data packet. The unified format is a structured text format or a binary serialization format. An example of fields in an interaction data packet includes: delivery identifier, key identifier, instruction text, object data, data type label, and trigger time. The data type label can be one or more of the following: text type, file type, and image type, where selected text corresponds to text type, file identifier corresponds to file type, screen area corresponds to image type, and cursor trajectory data corresponds to text type.

[0051] The interactive data packets adopt a unified format and are independent of the target delivery path determined subsequently; the differences between different delivery paths are handled in S4.

[0052] S4 determines the target delivery path, performs data type conversion, and then delivers the data.

[0053] See Figure 3 The client software first obtains the current availability status of each candidate delivery path for the target third-party intelligent agent.

[0054] The availability status of the interface call path is determined based on the connectivity detection result of the interface corresponding to the path: the client software initiates a probe request to the request address recorded in the adaptation rules, and if a response is received within a preset time period and the response status indicates availability, the availability status is available; otherwise, it is unavailable. The availability status of the component call path is determined based on the loading result of the software development kit (SDK) corresponding to the path: if the SSD has been installed on the electronic device and successfully loaded and initialized by the client software, the availability status is available; otherwise, it is unavailable. The availability status of the interface write path is determined based on whether the target process is running: if the target process is running, the availability status is available; if the target process is not running, the client software starts the target process, and after its target window appears, the availability status is determined to be available; if the target window does not appear within a preset time period, the availability status is determined to be unavailable.

[0055] The target delivery path is then determined in the following order: candidate delivery paths whose available status is unavailable are excluded; candidate delivery paths whose supported data types cover the labeled data type are prioritized over candidate delivery paths whose supported data types do not cover the labeled data type; candidate delivery paths in the same order are sorted according to a preset path priority; the path with the highest priority is selected as the target delivery path. The path priority is set according to actual needs, for example, interface call paths take precedence over component call paths, and component call paths take precedence over interface write paths; in other implementations, the path priority is adjusted based on the historical delivery confirmation results of each candidate delivery path.

[0056] For example, the object data obtained in this trigger is a file identifier, and the labeled data type is file type. The candidate delivery paths for the target third-party intelligent agent include an interface call path and a UI write path. The former supports text, file, and image types, while the latter only supports text types. When both paths are available, the interface call path, which covers the labeled data type, is ranked first and determined as the target delivery path. When the interface call path is unavailable, the UI write path is determined as the target delivery path. In this case, the labeled file type is not supported by the path, and data type conversion is initiated.

[0057] The data type conversion is performed according to the adaptation rules:

[0058] When the object data is a file identifier and the target delivery path does not support the file type, the file identifier is converted into the path text of the file on the electronic device; or the file is read and its characters are extracted, and the extracted characters are used as the conversion result. Which method is used is specified by the adaptation rules. For example, the former is used when the target third-party intelligent agent can read local files, and the latter is used when it cannot read local files.

[0059] When the object data is a screen area and the target delivery path does not support image types, the corresponding area image of the screen area is captured and uploaded to a preset storage service, and the returned resource location identifier is used as the conversion result; or text recognition is performed on the area image, and the recognized characters are used as the conversion result.

[0060] After the conversion is complete, the original data type, the converted data type, and the conversion method identifier involved in this conversion are recorded in the converted interaction data packet. The conversion method identifier indicates which conversion method was used. If there is no data type not supported by the target delivery path, the content of the interaction data packet remains unchanged, and the converted interaction data packet is the original interaction data packet.

[0061] The client software converts the transformed interactive data packet into a delivery format corresponding to the target delivery path according to the adaptation rules before delivery.

[0062] When the target delivery path is the interface call path, the instruction text, the object data, and the data type label are filled into the request body according to the field positions recorded in the adaptation rules. After attaching credentials according to the recorded authentication method, a request is sent to the recorded request address.

[0063] When the target delivery path is the component call path, the parameters are organized according to the input parameter format recorded in the adaptation rules, and the corresponding method provided by the software development kit is called.

[0064] When the target delivery path is the interface write path, see [link / reference]. Figure 4 The process proceeds sequentially as follows: Activate the target window belonging to the target process and bring it to the foreground; locate the control in the target window that has input focus and editable attributes, and determine whether it is in an input-enabled state based on the presence of a text insertion character. If it is in an input-enabled state, designate it as the target input control; if the control does not exist, traverse the control tree of the target window, score each candidate input control, and select the one with the highest score as the target input control and set its input focus; before writing, verify that the control currently having input focus is consistent with the target input control, and that the process to which the target window belongs is consistent with the target process; after verification, write the content to the target input control and perform a submit operation. The writing is achieved by setting the control content through the interface automation interface, or by placing the content in the clipboard and sending a paste command to the target input control; the submit operation is achieved by sending the Enter key to the target input control, or by clicking the submit control in the target window, the specific method of which is specified by the adaptation rules.

[0065] The score is accumulated based on the following factors: when the type identifier of the candidate input control belongs to a preset input control type, its score is increased; when the size of the candidate input control is greater than a preset size threshold, its score is increased; when the relative position of the candidate input control in the target window is within a preset area, its score is increased; when the prompt text of the candidate input control matches a preset prompt word set, its score is increased.

[0066] For the first third-party intelligent agent client, one set of values ​​for the above items is as follows: the input control type is a text editing box, with a weight of 3; the preset size threshold is a width not less than 40% of the target window width and a height not less than 24 pixels, with a weight of 2; the preset area is the lower third of the target window, with a weight of 2; the preset prompt word set includes "send message," "input question," and "how can I help you," with a weight of 3. For the second third-party intelligent agent client, whose input control is located at the top of the window, the preset area is set to the upper third of the target window, and the values ​​of the remaining items are the same as above. For the third third-party intelligent agent client, whose window contains both a search box and a message input box, both of which are text editing boxes, the preset prompt word set and the preset area are used to raise the score of the message input box higher than that of the search box.

[0067] Multiple interactive data packets corresponding to the same target process are delivered sequentially in a serial queue. After the delivery confirmation result of the previous interactive data packet is determined to be received or not received, the next interactive data packet is delivered. If the delivery confirmation result is that it cannot be confirmed, the serial queue does not release the next interactive data packet. At this time, the interactive data packet remains in a pending state and waits for user confirmation.

[0068] S5 Delivery Confirmation and Redelivery

[0069] See Figure 5 The client software obtains a delivery confirmation result based on the target delivery path, whereby the delivery confirmation result is either "received," "not received," or "cannot be confirmed."

[0070] When the target delivery path is the interface call path: if the received response message contains a receive identifier or a task identifier, it is determined that it has been received; if the received response message contains a rejection identifier or a call failure identifier, it is determined that it has not been received; if no response message is received within a preset time period, it is determined that it cannot be confirmed. One value of the preset time period is 5 seconds.

[0071] When the target delivery path is the component call path: if the callback event returned by the software development kit belongs to the preset receiving event type, it is determined to be received; if it belongs to the preset abnormal event type, it is determined to be not received; if there is no callback event within the preset time period, it is determined to be unconfirmed.

[0072] When the target delivery path is the interface writing path: after the submission operation is performed, the target input control is observed according to a preset sampling period, one of which is 200 milliseconds; if the content of the target input control changes from the written content to empty, it is determined that it has been received; if a new error message control appears in the target window, it is determined that it has not been received; if the content of the target input control is still consistent with the written content within a preset time, it is determined that it cannot be confirmed, one of which is 3 seconds.

[0073] If the delivery confirmation result is "not received" and there are other available candidate delivery paths for the target third-party intelligent agent, the delivery will be re-delivered through those other available candidate delivery paths. If there are no other available candidate delivery paths, the client software will output a prompt. If the delivery confirmation result is "cannot be confirmed," automatic re-delivery will not be performed. The client software will keep the interactive data packet in a pending state and output a prompt, and will only re-deliver after receiving a confirmation operation from the user via the mouse. The confirmation operation is to press the smart button or click the confirmation control in the prompt after it appears.

[0074] The client software assigns a delivery identifier to the interactive data packet, and carries the delivery identifier with each delivery; when re-delivering, the delivery identifier is used again, and the candidate delivery path used for this re-delivery is recorded in the delivery record corresponding to the delivery identifier. The delivery record includes the delivery time, the candidate delivery path used, and the delivery confirmation result obtained.

[0075] III. Output lighting effect status based on delivery confirmation results (optional implementation method)

[0076] In some implementations, the mouse further includes a light output unit, and the client software determines the light effect status based on the delivery confirmation result and sends it to the light output unit. The light effect statuses corresponding to "received", "not received", and "cannot be confirmed" are different from each other.

[0077] One set of lighting effect states is set as follows: when the delivery confirmation result is received, a solid green light is output for 1 second and then turns off; when not received, a red light flashes twice; when unable to confirm, a yellow breathing effect is output and remains until the interactive data packet leaves the pending state. When the target delivery path is the interface writing path and the target input control is being observed, a slow blue light flashes. In other embodiments, the difference between the lighting effect states is reflected in the different flashing frequencies while the colors are the same.

[0078] The client software adds an incrementing status number to the issued lighting effect status, and the lighting output unit discards lighting effect statuses with a status number no greater than that of the already executed status.

[0079] Users can thus know the status of the delivery without having to switch their attention to the interface of the third-party intelligent agent client.

[0080] IV. Electronic Devices and Storage Media

[0081] This application also provides an electronic device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the above-described method. The electronic device may further include a bus, a communication interface, and an input / output interface. The processor, the memory, and the communication interface are connected via the bus. The processor is a central processing unit, a digital signal processor, an application-specific integrated circuit (ASIC), or a programmable logic device (PLC). The memory includes random access memory (RAM) and non-volatile memory.

[0082] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method. The computer-readable storage medium includes media capable of storing program code, such as a magnetic disk, optical disk, read-only memory, and random access memory.

[0083] The above description is only a specific embodiment of this application. Any changes or substitutions made by those skilled in the art within the scope of the technology disclosed in this application should be covered within the protection scope of this application.

Claims

1. A multimodal data delivery method based on mouse button triggering, applied to an electronic device communicating with a mouse, wherein the mouse includes smart buttons and a microphone, and the electronic device runs client software, characterized in that, include: S1, obtain the access description record of the target third-party intelligent agent, the access description record includes the candidate delivery path of the third-party intelligent agent and the data types supported by each candidate delivery path and the corresponding adaptation rules; S2, responding to the press event of the smart button to start audio acquisition and screen object selection, responding to the release event of the smart button to terminate the audio acquisition and screen object selection, and obtaining the instruction text obtained by speech recognition of the acquired audio and the selected object data. S3, encapsulate the instruction text and the object data into an interactive data packet of a unified format, and label the data type to which the object data belongs; S4, obtain the availability status of each candidate delivery path, determine the target delivery path based on the marked data type, the data types supported by each candidate delivery path and the availability status; according to the adaptation rules, convert the object data in the interaction data packet corresponding to the data type not supported by the target delivery path into the data type supported by the target delivery path, and then convert the converted interaction data packet into the delivery form corresponding to the target delivery path before delivery; S5. Obtain a delivery confirmation result based on the target delivery path. The delivery confirmation result is either received, not received, or cannot be confirmed. If the delivery confirmation result is not received and there are other available candidate delivery paths, re-deliver through the other available candidate delivery paths. If the delivery confirmation result is cannot be confirmed, do not automatically re-deliver.

2. The method according to claim 1, characterized in that, The candidate delivery path includes at least two of the following: interface call path, component call path, and interface writing path. The access description record corresponding to the interface writing path also includes the target process. The data types supported by the interface call path and the component call path include text type, file type, and image type. The data types supported by the interface writing path include text type. The target third-party intelligent agent is a third-party intelligent agent pre-set in the client software, or a third-party intelligent agent determined based on the user's selection in the client software; the access description record is established through at least one of the following methods: active registration by the third-party intelligent agent, writing during software development kit installation, pre-setting in the client software, user configuration, and automatic discovery based on applications already installed on the electronic device; The determination of the target delivery path includes: excluding candidate delivery paths whose available status is unavailable; prioritizing candidate delivery paths whose supported data types cover the labeled data type over candidate delivery paths whose supported data types do not cover the labeled data type; sorting the candidate delivery paths in the same order according to a preset path priority; and selecting the one ranked first as the target delivery path.

3. The method according to claim 2, characterized in that, The process of obtaining the availability status of each of the candidate delivery paths includes: the availability status of the interface call path is determined based on the connectivity detection result of the interface corresponding to the path; the availability status of the component call path is determined based on the loading result of the software development kit corresponding to the path; the availability status of the interface writing path is determined based on whether the target process is in a running state. If the target process is not in a running state, the target process is started and the availability status is determined to be available after the target window to which the target process belongs appears.

4. The method according to claim 1, characterized in that, The object data includes at least one of the selected text, file identifier, cursor trajectory data, and screen area; the interactive data package also encapsulates the key identifier of the smart button. The screen object selection includes: recording the position sequence of the mouse cursor on the screen during the press of the smart button, and determining the selection range based on the start and end points of the position sequence; when the area of ​​the selection range is less than a preset area threshold, the selectable element at the start point is used as the selection result, otherwise the content within the selection range is used as the selection result. Based on the type of the host application to which the selection result belongs and the content format of the selection result, the selection result is determined to be one of the selected text, the file identifier, or the screen area, and is used as the object data; the cursor trajectory data includes the position sequence.

5. The method according to claim 4, characterized in that, The step of converting object data in the interactive data packet corresponding to data types not supported by the target delivery path into data types supported by the target delivery path includes: When the object data is the file identifier and the target delivery path does not support the file type, the file identifier is converted into the path text of the file on the electronic device, or into the text obtained by extracting the content of the file; When the object data is the screen area and the target delivery path does not support image types, the resource location identifier obtained after uploading the area image corresponding to the screen area, or the text obtained by performing text recognition on the area image, is used as the conversion result. The original data type, the converted data type, and the conversion method identifier involved in this conversion are recorded in the converted interactive data packet.

6. The method according to claim 2, characterized in that, When the target delivery path is the interface writing path, the delivery includes: activating the target window to which the target process belongs, determining the control with input focus and editable attributes in the target window as the target input control, if the control does not exist, traversing the control tree of the target window and scoring each candidate input control according to at least two of the control type identifier, size, relative position and prompt text, taking the one with the highest score as the target input control and setting its input focus, verifying that the control currently with input focus is consistent with the target input control and that the process to which the target window belongs is consistent with the target process before writing, writing after verification and performing a submission operation; Before determining the control with input focus and editable properties in the target window as the target input control, the method further includes determining whether the control is in an input state based on whether there is a text insertion character in the control. Only when the control is in an input state is it determined as the target input control. The scoring of each candidate input control includes: increasing the score of the candidate input control when the control's type identifier belongs to a preset type of input control; increasing the score of the candidate input control when the control's size is greater than a preset size threshold; increasing the score of the candidate input control when the control's relative position in the target window is within a preset area; and increasing the score of the candidate input control when the control's prompt text matches a preset prompt word set. Multiple interactive data packets corresponding to the same target process are delivered sequentially in a serial queue. After the delivery confirmation result of the previous interactive data packet is determined to be received or not received, the next interactive data packet is delivered. The step of obtaining the delivery confirmation result based on the target delivery path includes: after the submission operation is performed, observing the target input control at a preset sampling period, using the change of the content of the target input control from the written content to empty as the delivery confirmation result as the basis for determining that the delivery has been received, using the appearance of a new error message control in the target window as the basis for determining that the delivery has not been received, and determining that the content of the target input control is still consistent with the written content within a preset time period as unacceptable.

7. The method according to claim 2, characterized in that, The step of obtaining the delivery confirmation result based on the target delivery path includes: When the target delivery path is the interface call path, the receipt identifier or task identifier contained in the received response message is used as the basis for determining whether the delivery confirmation result has been received, and the rejection identifier or call failure identifier contained in the received response message is used as the basis for determining whether the delivery has not been received. If no response message is received within a preset time period, it is determined that the delivery cannot be confirmed. When the target delivery path is the component call path, the callback event returned by the software development kit belongs to a preset receiving event type as the criterion for determining whether it has been received, and belongs to a preset abnormal event type as the criterion for determining whether it has not been received. If there is no callback event within a preset time period, it is determined that it cannot be confirmed.

8. The method according to claim 1, characterized in that, The client software assigns a delivery identifier to the interactive data packet and carries the delivery identifier with each delivery; when re-delivering, the delivery identifier is reused and the candidate delivery path used in this re-delivery is recorded in the delivery record corresponding to the delivery identifier. When the delivery confirmation result is "cannot be confirmed", the client software keeps the interactive data packet in a pending state and outputs a prompt, and only re-delivers it after receiving the user's confirmation operation via the mouse.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images