Method for implementing visible-as-you-type based on components and auxiliary services, electronic device and storage medium

Abstract

This invention discloses a method, electronic device, and storage medium for implementing "visible and speakable" based on components and auxiliary services. The method includes: defining LyraDescWrapper and LyraIdWrapper components on the SDK access side, embedding the component's id and desc as JSON tags into a Compose component; on the server side, using the UiHelperProxy class, determining whether to use Compose parsing mode based on the event package name; if in Compose mode, extracting JSON tag data from the ContentDescription of AccessibilityNodeInfo and parsing the corresponding id and description information; matching the parsed tag data with the voice input command and triggering the corresponding AccessibilityNodeInfo to perform an operation. This application's method uses Compose parsing, reducing intrusion into existing "visible and speakable" mechanisms and minimizing modifications. Through tag-based parameter settings, data can be added to components more flexibly and extensibly. In the future, more Compose components can be encapsulated to support more types of "visible and speakable" effects, and the application's dependency on Conifg files can be reduced.

Description

Technical Field

[0001] This invention belongs to the field of speech processing technology, and particularly relates to a method for implementing "see-and-speak" based on components and auxiliary services, as well as an electronic device and storage medium. Background Technology

[0002] Similar technologies include the "see-and-talk" functionality based on a view architecture. This technology uses auxiliary services to obtain page node elements, parses information such as text, contentDescription, and id of the node elements, caches the corresponding text information, and when voice is triggered, finds the corresponding node through the cached text keyword information, thereby enabling operations such as clicking the node.

[0003] The inventors discovered that the view-based "see-and-say" functionality cannot be implemented in apps based on Compose, such as clicking on lists configured in local configuration files or matching statements to specific views. Furthermore, direct text matching lacks extensibility and is not as flexible and extensible as tag-based parsing.

[0004] The inventors also discovered that the existing technology lacks the completeness of the Tianqin "See and Say" solution, including generalization functionality, keyword configuration for specific IDs, list selection matching, etc., and that after the customer's app is refactored using Compose, when it needs to handle the corresponding matching rules based on the ID in the local "See and Say" solution, some methods under the ordinary view system cannot be used. Summary of the Invention

[0005] The embodiments of the present invention are intended to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a visible-to-speak implementation method based on components and auxiliary services, comprising: defining LyraDescWrapper and LyraIdWrapper components on the SDK access end, embedding the component's id and desc in JSON format tags into the Compose component; on the server end, determining whether to use Compose parsing mode based on the event package name through the UiHelperProxy class, if it is in Compose mode, extracting JSON tag data from the ContentDescription of AccessibilityNodeInfo, and parsing out the corresponding id and description information; matching the parsed tag data according to the voice input command, and triggering the corresponding AccessibilityNodeInfo to perform an operation.

[0007] Secondly, embodiments of the present invention also provide a system based on components and auxiliary services where visibility is directly audible. This system is used to implement the method described in the first aspect, and includes: an application-side SDK for integrating LyraDescWrapper and LyraIdWrapper components to achieve tag-based encapsulation of the Compose component; a server-side parsing module for adapting to both View and Compose modes via the UiHelperProxy class, parsing tag data, and executing voice commands; and a local configuration file for storing application package names registered in Compose mode and their corresponding parsing rules.

[0008] Thirdly, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described component- and auxiliary service-based visible-and-sayable implementation methods of the present invention.

[0009] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, the execution instructions being readable and executable by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform any of the above-described component- and auxiliary service-based implementation methods of the present invention.

[0010] Fifthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described component- and auxiliary service-based visible-and-sayable implementation methods.

[0011] This invention utilizes the Compose method for parsing, minimizing intrusion into existing visible-to-speak effects and achieving minimal modifications. Through tag-based parameter settings, data can be added to components more flexibly and extensibly. In the future, more Compose components can be encapsulated to support a wider variety of visible-to-speak effects, and the application's dependency on Conifg files can be reduced. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating one embodiment of the visible-and-sayable implementation method of the present invention based on components and auxiliary services; Figure 2 A flowchart illustrating the process of a visible-and-say implementation method based on components and auxiliary services provided by the present invention; Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0016] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, elements, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0017] In this invention, terms such as "module," "device," and "system" refer to relevant entities applied to a computer, such as hardware, combinations of hardware and software, software, or software in execution. More specifically, for example, an element can be, but is not limited to, a process running on a processor, a processor, an object, an executable element, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be an element. One or more elements may be in an execution process and / or thread, and elements may be localized on a single computer and / or distributed across two or more computers, and may be run on various computer-readable media. Elements can also communicate via local and / or remote processes based on signals having one or more data packets, for example, signals from data interacting with another element in a local system, a distributed system, and / or interacting with other systems via signals over a network on the Internet.

[0018] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0019] This invention provides a visible-as-talkable implementation method based on components and auxiliary services, which can be applied to electronic devices. Electronic devices can be computers, servers, or other electronic products, and this invention does not limit them.

[0020] Please refer to Figure 1 This illustrates a visible-and-say implementation method based on components and auxiliary services provided by an embodiment of the present invention.

[0021] like Figure 1 As shown, in step 101, by defining LyraDescWrapper and LyraIdWrapper components on the SDK access side, the id and desc of the components are embedded into the Compose component in JSON format. In step 102, on the server side, the UiHelperProxy class determines whether to use the Compose parsing mode based on the event package name. If it is in Compose mode, the JSON tag data is extracted from the ContentDescription of AccessibilityNodeInfo, and the corresponding id and description information are parsed out. In step 103, the parsed tag data is matched according to the voice input command, and the corresponding AccessibilityNodeInfo is triggered to perform the operation.

[0022] In this embodiment, for step 101, LyraDescWrapper and LyraIdWrapper components are defined on the SDK access side, and the component's id and desc are embedded into the Compose component using JSON format tags. For example, on the SDK access side: two components, LyraDescWrapper and LyraIdWrapper, are defined. This component receives input parameters id, desc, and child components, and internally tags the id, desc, etc., using JSON format tags. Taking advantage of the fact that nested use of Compose does not affect performance, users only need to directly apply LyraDescWrapper or LyraIdWrapper to the original component on the app side to achieve configuration. The SDK (Software Development Kit) access side usually refers to the process of integrating a third-party SDK on the client side for quickly implementing specific functions. LyraDescWrapper can be used to wrap UI elements that require additional descriptive information, such as adding help text next to form input. LyraIdWrapper can be used to provide a unique ID for an element, or to associate labels and input fields in a form to ensure accessibility. JSON-formatted tagging refers to attaching metadata to components, such as storing `id` and `desc` as JSON strings in component properties, or serializing the component structure into JSON. Compose is the latest UI rendering method for Android.

[0023] Next, for step 102, on the server side, the UiHelperProxy class determines whether to use the Compose parsing mode based on the event package name. If it is in Compose mode, it extracts JSON tag data from the ContentDescription of AccessibilityNodeInfo and parses out the corresponding ID and description information. On the service parsing side: in order to be compatible with the "see-is-say" service under the current view system, all view-related methods used in the auxiliary service (such as findViewById) are abstracted and encapsulated in UiHelperProxy. In this class, there are functions such as clickListItem, findNodesById, isExistListWidget, findNodeByDescOrText. By passing in the packageName, it obtains from the local configuration file whether the package name of the event to be implemented is registered as compose. If it is, it is handed over to ComposeUiHelper; if not, it is handed over to ViewUiHelper. In ComposeUiHelper, for contentDesc and id, a JSON string is uniformly obtained from the ContentDesc of AccessibilityNodeInfo, and the corresponding id and desc are obtained according to the JSON data defined by LyraDescWrapper and LyraIdWrapper in the SDK, thus realizing the tag-based configuration and parsing of components.

[0024] Finally, for step 103, the parsed tag data is matched according to the voice input command, triggering the corresponding AccessibilityNodeInfo to perform the operation. AccessibilityNodeInfo is a structural unit scanned in the service. AccessibilityNodeInfo is a tree structure that is independent of the specific UI rendering.

[0025] The method in this application uses Compose to parse data, reducing intrusion into existing visible-to-speak effects and minimizing modifications. Through tag-based parameter settings, data can be added to components more flexibly and extensibly. In the future, more Compose components can be encapsulated to support a wider variety of visible-to-speak effects, and the application's dependency on Compose files can be reduced.

[0026] It should be noted that existing technologies lack the completeness of the Tianqin "Visible and Speakable" solution, including generalization capabilities, keyword configuration for specific IDs, and list selection matching statements. This application extends the existing Tianqin "Visible and Speakable" solution for compatibility with Compose applications. After the client's app was refactored using Compose, some methods from the normal view system could not be used when handling local matching rules configured based on IDs in "Visible and Speakable". This application, after referring to the Android official documentation, discovered that AccessibilityNodeInfo in the Accessibility service is a tree structure independent of specific UI rendering. Compose also supports adding ContentDescription. Leveraging the advantage of nested Compose components that do not cause redrawing, this application chose to use nested custom Compose components to add tagged data, with Tianqin parsing the corresponding structure. This application uses a formatted Compose component encapsulated on the app side to tag specific parameters, and on the Tianqin side, it parses the corresponding tags according to the data structure standard.

[0027] In some optional embodiments, the LyraDescWrapper and LyraIdWrapper components wrap the target Compose component in a nested manner, converting the input id and desc parameters into JSON strings and storing them in the component's ContentDescription property. In Compose, ContentDescription is set via Modifier.semantics and can store strings. The id and desc are combined into a JSON string and stored. LyraDescWrapper and LyraIdWrapper need to wrap the target component and modify its modifiers to add semantic information. In Compose, the target component can be wrapped using a layout or a custom layout and then applied.

[0028] In some optional embodiments, the UiHelperProxy class includes the following functional modules: It determines whether the current application is in Compose mode based on the package name from the local configuration file; if in Compose mode, it calls the ComposeUiHelper module to parse the JSON tag data in ContentDescription; if not in Compose mode, it calls the ViewUiHelper module to directly obtain the ID and text information of the View component. UiHelperProxy can cache the package name of the currently occurring event. The ComposeUiHelper module recursively traverses page nodes, extracts the ContentDescription field from AccessibilityNodeInfo, parses the JSON-formatted tag data to obtain the component's ID and description information, and matches it with voice input commands. UiHelperProxy caches the package name of the currently occurring event. Recursively traversing page nodes, UiHelperProxy records text / contentDesc and ID based on the package name, using different methods depending on whether it's view or compose; compose uses tag-based parsing, while view directly retrieves the data. UiHelperProxy is a design pattern or utility class used to decouple UI components from underlying business logic, state management, or complex operations. It uses a "proxy" mechanism to extract auxiliary functions (such as data loading, event handling, and state synchronization) from the UI layer. Using UiHelperProxy allows for more efficient code organization and improves the maintainability and scalability of complex UI projects.

[0029] In some optional embodiments, during the event callback phase, the AccessibilityActionService filters package name events not registered in the configuration file; it caches the current event package name and page node information to form a TextBean data set; during voice input, it matches the TextBean data with instructions to locate the target node and execute the operation. For example, after caching the page's textbean information, when voice input occurs, it intercepts the dialogue, obtains the input, compares the input with the cached textbean information, and determines whether the "Speak Now" action is required. If action is required, it finds the corresponding TextBean and hands it over to the AccessibilityActionService for execution. The AccessibilityActionService traverses the node tree to find the corresponding Node; the AccessibilityNodeInfo executes the corresponding action. AccessibilityActionService is an accessibility service, a special function provided by Android for people with disabilities.

[0030] In some optional embodiments, the JSON tag data includes at least the following fields: component unique identifier id, description text desc, component type type, and supports dynamically expanding custom fields.

[0031] It should be noted that the method in this application uses a more standardized and simpler approach by encapsulating the Compose component on the SDK side to add corresponding data to the Compose component on the app side. The Lyra side only determines whether to use the Compose method for parsing based on the package name during parsing—reducing intrusion into the existing "visible and audible" functionality and minimizing modifications. Tag-based parameter settings allow for more flexible and extensible data addition to components. In the future, more Compose components can be encapsulated to support more types of "visible and audible" effects and reduce the Lyra side's dependency on Conifg files.

[0032] This application also provides a visible-to-talk system based on components and auxiliary services. The system is used for the above-mentioned visible-to-talk implementation method based on components and auxiliary services. The system includes: an application SDK for integrating LyraDescWrapper and LyraIdWrapper components to implement the tag-based encapsulation of the Compose component. The server-side parsing module is used to adapt to both View and Compose modes through the UiHelperProxy class, parse tag data and execute voice commands; the server-side parsing module also includes a node traversal module, which is used to recursively access the page node tree and extract tag data; The execution module is used to invoke the input operations of AccessibilityNodeInfo based on the matching results.

[0033] The local configuration file stores the application package names registered in Compose mode and their corresponding resolution rules.

[0034] It should be noted that this application also provides an alternative version that completely encapsulates and parses the List type component through standardization. The List type component completely eliminates the need for configuration on the Tianqin client, allowing for custom configuration on the app side. However, this alternative version is not compatible with existing List type components.

[0035] The overall design concept and principle of this application are as follows: The Compose component, encapsulated and formatted on the app side, is used to tag specific parameters. On the Tianqin side, the corresponding tags are parsed according to data structure standards. The solution consists of an SDK access end and a service parsing end. The SDK access point defines two components, LyraDescWrapper and LyraIdWrapper. These components receive input parameters id, desc, and child components. Internally, they tagged the id, desc, and other content using JSON format. Leveraging the performance-free nature of nested Compose components, users can simply apply LyraDescWrapper or LyraIdWrapper directly to the original components in the app to achieve the configuration.

[0036] Service parsing side: To ensure compatibility with the current view-based "visible and speakable" service architecture, all view-related methods used in auxiliary services (such as `findViewById`) are abstracted and encapsulated in `UiHelperProxy`. This class includes functions like `clickListItem`, `findNodesById`, `isExistListWidget`, and `findNodeByDescOrText`. Using the passed `packageName`, it retrieves whether the package name corresponding to the event to be implemented is registered as `compose` from the local configuration file. If so, it passes the request to `ComposeUiHelper`; otherwise, it passes it to `ViewUiHelper`. In `ComposeUiHelper`, for `contentDesc` and `id`, it uniformly retrieves the JSON string from `AccessibilityNodeInfo`'s `ContentDescrition` and, based on the JSON data defined in the SDK's `LyraDescWrapper` and `LyraIdWrapper`, obtains the corresponding configured `id` and `desc`, thus achieving tag-based configuration and parsing for components.

[0037] Please refer to Figure 2 The document presents a flowchart illustrating the implementation of the visible-and-sayable method based on components and auxiliary services of the present invention.

[0038] like Figure 2 As shown: Step 1: When the AccessibilityActionService event callback occurs, analyze the package name of the event, compare it with the package names registered in the configuration file, and filter the current event; Step 2: UiHelperProxy caches the package name of the currently occurring event; Step 3: Recursively traverse the page nodes. UiHelperProxy records text / contentDesc and id in different ways depending on the package name, view, or compose. The compose method uses tag-based parsing, while the view method directly retrieves the data. Step 4: Cache the textbean information on the page; Step 5: When voice input occurs, intercept the conversation and obtain the input; Step 6: Based on the cached textbean information, compare the input and determine whether visibility needs to be enabled; Step 7: If execution is required, locate the corresponding TextBean and hand it over to the AccessibilityActionService for execution; Step 8: AccessibilityActionService traverses the node tree to find the corresponding Node; Step 9: AccessibilityNodeInfo executes the corresponding action.

[0039] This application's method uses Compose to parse components, minimizing intrusion into existing visible-to-speak effects and achieving minimal modifications. Through tag-based parameter settings, data can be added to components more flexibly and extensibly. In the future, more Compose components can be encapsulated to support a wider variety of visible-to-speak effects, and the application's dependency on Conifg files can be reduced.

[0040] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0041] In some embodiments, the present invention provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by an electronic device (including but not limited to a computer, server, or network device, etc.) to perform any of the above-described component- and auxiliary service-based implementation methods of the present invention.

[0042] In some embodiments, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform any of the above-described component- and auxiliary service-based visible-and-sayable implementation methods.

[0043] In some embodiments, the present invention also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a visible-and-sayable implementation method based on components and auxiliary services.

[0044] Figure 3 This is a schematic diagram of the hardware structure of an electronic device that performs a visible-and-sayable implementation method based on components and auxiliary services, according to another embodiment of this application. Figure 3 As shown, the device includes: One or more processors 310 and memory 320, Figure 3 Take the 310 processor as an example.

[0045] The device that implements the visible-as-talkable implementation method based on components and auxiliary services may further include: an input device 330 and an output device 340.

[0046] The processor 310, memory 320, input device 330, and output device 340 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0047] Memory 320, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the component- and auxiliary service-based visible-and-tell implementation method in the embodiments of this application. Processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in memory 320, thereby implementing the component- and auxiliary service-based visible-and-tell implementation method of the above-described method embodiments.

[0048] Memory 320 may include a program storage area and a data storage area. The program storage area may store an operating system and applications required for at least one function. The data storage area may store data created based on the use of the component- and auxiliary services-based visible-and-tellable implementation device. Furthermore, memory 320 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 320 may optionally include memory remotely located relative to processor 310, which can be connected to the component- and auxiliary services-based visible-and-tellable implementation device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0049] Input device 330 can receive input digital or character information and generate signals related to user settings and function control of the visible-and-talkable implementation device based on components and auxiliary services. Output device 340 may include display devices such as a display screen.

[0050] The one or more modules are stored in the memory 320, and when executed by the one or more processors 310, they execute the visible and speakable implementation method based on components and auxiliary services in any of the above method embodiments.

[0051] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0052] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones, multimedia phones, feature phones, and low-end phones, etc.

[0053] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, etc.

[0054] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0055] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0056] (5) Other electronic devices with data interaction functions.

[0057] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0058] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visible-as-talkable implementation method based on components and auxiliary services, comprising: By defining LyraDescWrapper and LyraIdWrapper components in the SDK access end, the component's id and desc are embedded into the Compose component in JSON format. On the server side, the UiHelperProxy class determines whether to use the Compose parsing mode based on the event package name. If it is in Compose mode, the JSON tag data is extracted from the ContentDescription of AccessibilityNodeInfo, and the corresponding ID and description information are parsed out. The corresponding AccessibilityNodeInfo is triggered to perform an operation based on the parsed tag data matched with the voice input command.

2. The method according to claim 1, characterized in that, The LyraDescWrapper and LyraIdWrapper components wrap the target Compose component in a nested manner, and convert the input id and desc parameters into JSON strings, which are then stored in the component's ContentDescription property.

3. The method according to claim 1, characterized in that, The UiHelperProxy class includes the following functional modules: Determine whether the current application is in Compose mode based on the package name from the local configuration file; If in Compose mode, the ComposeUiHelper module is called to parse the JSON tag data in ContentDescription; If it is not in Compose mode, the ViewUiHelper module is called to directly obtain the ID and text information of the View component.

4. The method according to claim 3, characterized in that, The ComposeUiHelper module recursively traverses page nodes, extracts the ContentDescription field from AccessibilityNodeInfo, parses JSON-formatted tag data to obtain the component's ID and description information, and matches it with voice input commands.

5. The method according to claim 1, characterized in that, It also includes the following steps: During the event callback phase, AccessibilityActionService filters out package name events that are not registered in the configuration file; Cache the current event package name and page node information to form a TextBean data collection; When using voice input, the target node is located and the operation is performed based on the TextBean data matching instructions.

6. The method according to claim 1, characterized in that, The JSON tag data must include at least the following fields: component unique identifier id, description text desc, component type type, and supports dynamically extended custom fields.

7. A visible-and-talkable system based on components and auxiliary services, for use with the method of any one of claims 1-6, characterized in that, include: The application-side SDK is used to integrate LyraDescWrapper and LyraIdWrapper components, enabling the tag-based encapsulation of Compose components; The server-side parsing module is used to adapt to both View and Compose modes through the UiHelperProxy class, parse tag data and execute voice commands; The local configuration file stores the application package names registered in Compose mode and their corresponding resolution rules.

8. The system according to claim 7, characterized in that, The server-side parsing module also includes: The node traversal module is used to recursively access the page node tree and extract tagged data; The execution module is used to invoke the input operations of AccessibilityNodeInfo based on the matching results.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 6.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images