Electronic equipment control method and system

Abstract

The invention provides an electronic equipment control method and system.The method is applied to an equipment terminal and comprises the steps that a voice control instruction from a control terminal is received, the equipment terminal establishes communication connection with the control terminal by establishing a direct connection group, and the voice control instruction carries request type identification and request data; according to the request type identifier and a preset first mapping relationship, determining a target request type of the voice control instruction, the first mapping relationship being a one-to-one correspondence relationship between the request type identifier and the request type; according to the target request type and a preset second mapping relationship, determining a target processing class of the voice control instruction, the second mapping relationship being a one-to-one correspondence relationship between the request type and the processing class; calling the target processing class, and operating the electronic equipment according to the request data. Through the microphone-free voice adaptation scheme, the voice control of the electronic equipment can be realized without depending on a microphone, so that the equipment cost can be reduced, and the user experience is not influenced.

Description

Technical Field

[0001] The present application relates to the technical field of device voice control, and in particular to a method and system for controlling an electronic device. Background Art

[0002] With technological advancements and growing user demands, the intelligentization of electronic devices has become a trend. Within this trend, voice interaction has become an essential component of electronic device intelligence. In smart devices, users can control various device functions, such as navigation, entertainment systems, and phone calls, through voice commands, greatly enhancing the user experience.

[0003] In the related art, in order to realize the voice interaction function of electronic devices, they need to be equipped with high-quality microphone devices to ensure the accuracy and response speed of voice recognition. However, for electronic devices that do not install microphones to reduce costs, or whose microphones malfunction or perform poorly, this will directly affect the user experience of smart devices and reduce user satisfaction. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present application discloses an electronic device control method and system for solving the technical problem of poor user experience in intelligent operation of electronic devices in the prior art.

[0005] In the first aspect, the present application provides an electronic device control method, which is applied to a device terminal, wherein the device terminal is independent of the electronic device or integrated in the electronic device, and the method includes: receiving a voice control instruction from the control terminal, wherein the device terminal establishes a communication connection with the control terminal by creating a direct connection group, and the voice control instruction carries a request type identifier and request data; determining a target request type of the voice control instruction based on the request type identifier and a preset first mapping relationship, wherein the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type; determining a target processing class of the voice control instruction based on the target request type and a preset second mapping relationship, wherein the second mapping relationship is a one-to-one correspondence between the request type and the processing class; calling the target processing class, and operating the electronic device according to the request data.

[0006] In one embodiment of the present application, the target request type is one of a page operation request, an application start request, and a semantic search request; the target processing class is one of a page information processing class, an application start processing class, and a semantic search processing class.

[0007] In one embodiment of the present application, the target processing class is called to operate the electronic device according to the request data, including: if the target request type is a page operation request, the page information processing class is obtained, and the page information processing class includes an accessibility service class and a page root element processing class, and the accessibility service class defines a root element acquisition method and implements a page root element listening interface; calling the page root element processing class to create a root element listening instance, and calling the root element acquisition method to obtain the page root element, and calling back the page root element and assigning it to the root element listening instance; operating the electronic device according to the assigned root element listening instance and the request data.

[0008] In one embodiment of the present application, operating the electronic device based on the assigned root element monitoring instance and the request data includes: calling the find element method in the page root element processing class, and traversing the page root element and the child elements corresponding to each of the page root elements in the find element method; if there is a target element that matches the request data, triggering a click event of the target element to operate the electronic device.

[0009] In one embodiment of the present application, the method for determining the target element includes: comparing an index parameter with a preset parameter threshold, wherein the index parameter is carried in the voice control instruction, and the index parameter is an identifier that uniquely identifies the target element indicated by the request data; if the index parameter is greater than the parameter threshold, then in the process of traversing the page root element and the child elements, the number of times the element that matches the request data is traversed is recorded, and when the number matches the index parameter, the element corresponding to the number is determined as the target element.

[0010] In one embodiment of the present application, the target processing class is called to operate the electronic device according to the request data, including: if the target request type is a semantic search request, obtaining a semantic search processing class; calling the semantic search processing class to obtain a data parsing method and an activity startup method; using the data parsing method to parse the request data to obtain an activity instance, which includes a target application package name and an activity name; passing the activity instance to the activity startup method to trigger an activity startup event to operate the electronic device.

[0011] In one embodiment of the present application, before receiving the voice control instruction from the control terminal, it also includes: detecting whether the electronic device has a voice capture and recognition function; if the electronic device does not have the voice capture and recognition function, sending a voice control instruction through the control terminal to control the electronic device; if the electronic device has the voice capture and recognition function, detecting whether the voice capture and recognition function has a fault, and if the voice capture and recognition function has a fault, sending a voice control instruction through the control terminal to control the electronic device.

[0012] In a second aspect, the present application provides an electronic device control method, which is applied to a control terminal, the method comprising: if a communication connection with the device terminal is successfully established, capturing user audio data and converting the user audio data into text data, the communication connection being established through a direct connection group created by the device terminal, and the device terminal being independent of the electronic device or integrated into the electronic device; parsing the text data into formatted data, the formatted data including a request type identifier and request data, the request type identifier being determined based on a request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; encapsulating the formatted data into a voice control instruction, the voice control instruction carrying the request type identifier and the request data; and sending the voice control instruction to the device terminal so that the device terminal operates the electronic device according to the voice control instruction.

[0013] In one embodiment of the present application, the request type is one of a page operation request, an application start request, and a semantic search request; if the request type is a page operation request, the formatted data also includes an index parameter, which is an identifier that uniquely identifies the target element referred to by the request data.

[0014] In a third aspect, the present application provides an electronic device control system, the system comprising: a device terminal for creating a direct connection group, the device terminal being independent of the electronic device or integrated in the electronic device; a control terminal for establishing a communication connection with the device terminal according to the direct connection group; if the communication connection is successfully established, capturing user audio data and converting the user audio data into text data; parsing the text data into formatted data, the formatted data comprising a request type identifier and request data, the request type identifier being determined based on a request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; The formatted data is encapsulated into a voice control instruction, which carries the request type identifier and the request data; the voice control instruction is sent to the device terminal; the device terminal is also used to receive the voice control instruction from the control terminal; according to the request type identifier in the voice control instruction and the first mapping relationship, the target request type of the voice control instruction is determined; according to the target request type and a preset second mapping relationship, the target processing class of the voice control instruction is determined, and the second mapping relationship is a one-to-one correspondence between the request type and the processing class; the target processing class is called to operate the electronic device according to the request data in the voice control instruction.

[0015] As described above, the electronic device control method and system provided by the embodiments of the present application have the following beneficial effects: The device terminal establishes a communication connection with the control terminal by creating a direct connection group to receive a voice control instruction from the control terminal. The voice control instruction carries a request type identifier and request data. Then, based on the request type identifier and a preset first mapping relationship, the target request type of the voice control instruction is determined. The first mapping relationship is a one-to-one correspondence between the request type identifier and the request type. Then, based on the target request type and a preset second mapping relationship, the target processing class of the voice control instruction is determined. The second mapping relationship is a one-to-one correspondence between the request type and the processing class. Finally, the target processing class is called to operate the electronic device according to the request data. The control terminal captures the user voice and controls the electronic device by receiving the voice control instruction transmitted by the control terminal. In addition, for different voice control requests, the device terminal is configured with different processing classes, which can meet the precise execution of different voice control requests. In this way, through the microphone-free voice adaptation solution, voice control of electronic devices is realized without relying on a microphone, which not only reduces the device cost but also improves the user experience.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings: Figure 1 This is a schematic diagram of an implementation environment of an electronic device control system shown in an exemplary embodiment of the present application; Figure 2 is a schematic diagram of an implementation environment of another electronic device control system shown in an exemplary embodiment of the present application; Figure 3 is a flow chart of a method for controlling an electronic device according to an exemplary embodiment of the present application; Figure 4 This is a page information processing class diagram shown in an exemplary embodiment of the present application; Figure 5 This is a diagram of an application startup process shown in an exemplary embodiment of the present application; Figure 6 is a semantic search processing class diagram shown in an exemplary embodiment of the present application; Figure 7 This is a device terminal class diagram shown in an exemplary embodiment of the present application; Figure 8 is a flowchart of another electronic device control method shown in an exemplary embodiment of the present application; Figure 9 This is a control terminal class diagram shown in an exemplary embodiment of the present application; Figure 10 is a flowchart of a specific electronic device control method shown in an exemplary embodiment of the present application; Figure 11 It is a block diagram of an electronic device control system shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will describe the embodiments of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand the other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for the purpose of illustrating the present application and are not intended to limit the scope of protection of the present application.

[0019] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0020] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0021] Smart devices require high-quality microphones to ensure accurate and responsive voice recognition, enhancing the user experience. However, the inventors have discovered that microphone configuration and maintenance pose significant challenges in terms of cost-effectiveness. On the one hand, installing high-quality microphones increases device hardware costs and reduces profit margins. On the other hand, microphone failure or poor performance directly impacts the user experience of smart devices, reducing user satisfaction.

[0022] Therefore, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of an electronic device control system according to an exemplary embodiment of the present application. Figure 1 As shown, the implementation environment includes a control terminal 110, a device terminal 120 and an electronic device 130, wherein the control terminal 110 is independent of the electronic device 130, and the device terminal 120 is integrated into the electronic device. The control terminal 110 and the device terminal 120 are used to implement voice control of the electronic device 130. In addition, see Figure 2 , Figure 2 FIG. 1 is a schematic diagram of another implementation environment of an electronic device control system shown in an exemplary embodiment of the present application. Figure 2As shown, the implementation environment includes a control terminal 210, a device terminal 220, and an electronic device 230, wherein the control terminal 210 is independent of the electronic device 230, and the device terminal 220 is also independent of the electronic device 130. The control terminal 110 and the device terminal 120 are used to implement voice control of the electronic device 130. In the electronic device control system, the control terminal captures the user voice, parses and encapsulates the user voice, and transmits the voice control instructions to the device terminal to implement voice control of the electronic device. The control terminal can provide support for diverse user voices. In the device terminal, different processing classes are configured for voice control instructions of different request types, which can meet the precise execution of different voice control requests. In this way, through the microphone-free voice adaptation solution, voice control of the electronic device is implemented without relying on a microphone, which not only reduces costs but also improves user experience.

[0023] Electronic devices can be appliances such as televisions, refrigerators, washing machines, air conditioners, and robot vacuums. The use scenarios of these appliances are not limited, such as household appliances and automotive appliances. They can also be entertainment electronic devices such as audio systems, players, and game consoles. They can also be industrial electronic devices or vehicle systems. Electronic devices can have or not have voice capture and recognition capabilities. The control terminal can be a mobile phone, tablet, laptop computer, tablet computer, smart watch, smart bracelet, or other device with voice capture and recognition capabilities.

[0024] For example, in Figure 1 In the implementation environment, the control terminal 110 may be a mobile phone, the device terminal 120 may be a vehicle-mounted terminal, and the electronic device 130 may be a vehicle-mounted system. That is, the user's voice is captured by the mobile phone, and the user's voice is parsed and packaged, and the voice control command is transmitted to the vehicle-mounted terminal to realize the voice control of the vehicle-mounted system. Figure 2 In the implementation environment, the control terminal 210 can be a mobile phone, the device terminal 220 can be a vehicle-mounted terminal, and the electronic device 230 can be a vehicle-mounted air conditioner. That is, the user voice is captured by the mobile phone, and the user voice is parsed and packaged, and the voice control instructions are transmitted to the vehicle-mounted terminal to realize voice control of the vehicle air conditioner.

[0025] See Figure 3 , Figure 3 This is a flowchart of an electronic device control method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 or Figure 2 The implementation environment shown is shown. It should be understood that the method can also be applied to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0026] In addition, the electronic device control method can be applied to device terminals under various operating systems, including but not limited to device terminals under the Android operating system.

[0027] like Figure 3 As shown, in an exemplary embodiment, the electronic device control method includes at least steps S310 to S340, which are described in detail as follows: Step S310: receiving a voice control instruction from the control terminal, and the device terminal establishes a communication connection with the control terminal by creating a direct connection group. The voice control instruction carries a request type identifier and request data.

[0028] Among them, the voice control instruction refers to the instruction for the control terminal to parse and encapsulate the captured user voice instruction into formatted data. The voice control instruction includes a request type identification field and a request data field. The request type identification field describes the request type identification, and the request type identification represents the request type of the voice control instruction; the request data field describes the request data, and the request data represents the request purpose of the voice control instruction, that is, the user intention.

[0029] Exemplarily, "action":"..." is used as the request type identification field, and "data":"..." is used as the request data field.

[0030] In a possible embodiment, the voice control instruction also includes a communication identification field, which is used to identify a communication between the device terminal and the control terminal. This makes it easier to track the communication connection process between the two and the flow path of interactive information, monitor the entire life cycle of the communication, and facilitate problem troubleshooting and log analysis.

[0031] Exemplarily, "requestId":"..." is used as the communication identification field.

[0032] When two devices connect via P2P (peer-to-peer) communication, two primary roles are defined: the P2P Group Owner (GO) and the P2P Group Client (GC). A device creates a direct group connection, acting as the GO, equivalent to a server, while the other connected device acts as the GC, equivalent to a client. In this embodiment, the device acts as the GO, or server, and the control terminal acts as the GC, or client. Other devices can join the group by connecting to the GO device, but cannot connect directly to the GC device.

[0033] In one embodiment, a method for establishing a communication connection includes: obtaining and calling a first direct connection management class, creating a direct connection group, so that the control terminal searches for and accesses the group owner of the direct connection group, where the group owner is a device terminal; creating a server management class, where the server management class includes member variables, a server socket, a thread pool object, and a server startup method, where a connection acceptance method is defined; in the thread pool, calling the server startup method to create a socket instance, executing the connection acceptance method, and monitoring the access of the control terminal; if the access of the control terminal is monitored, establishing a communication connection.

[0034] Among them, the first direct management class is pre-configured on the device terminal.

[0035] For example, the first direct connection management class is the Wi-Fi direct connection management class, WifiP2pManager. This class is the WifiP2pManager for Wifi P2P (Wi-Fi Peer-to-Peer) technology (for example, in an Android device, this is the Android Wifi P2P technology, i.e., the WifiP2pManager for Android Wi-Fi peer-to-peer technology). The direct connection group is the Wifi P2p Group (Wi-Fi Peer-to-Peer Group).

[0036] Exemplarily, a method for creating a direct connection group of a device terminal is to obtain the first direct connection management class WifiP2pManager according to the direct connection management class acquisition method provided by the operating system, and call the initialize() method of the first direct connection management class WifiP2pManager for initialization, create a channel Channel, obtain a channel instance, that is, a Channel instance, and then call the direct connection group creation method createGroup() of the first direct connection management class WifiP2pManager and pass in the Channel instance to create a direct connection group and become the group owner.

[0037] In this embodiment, the server-side management class ServerManager is used to manage connections to device terminals. This class includes member variables: ① a server-side socket ServerSocket, used to transfer data with the control terminal; and ② a thread pool object ExecutorService, used to provide child threads for sending and receiving data with the control terminal. This class also defines methods: ① a server-side startup method, startServer(), used to start the device terminal waiting for a connection from the control terminal. Specifically, the server-side startup method, startServer(), defines a connection acceptance method, serverSocket.accept(), used to accept access from the device terminal and establish a communication connection.

[0038] In this embodiment, after the direct connection group is successfully created, a server management instance, i.e., a ServerManager instance, is first created to call the server startup method startServer() to create a socket instance, i.e., a ServerSocket instance. In the ServerSocket instance, the port number is specified. Then, the server startup method startServer() calls the connection acceptance method serverSocket.accept() to wait for the access of the control terminal. If the control terminal access is monitored, a communication connection is established.

[0039] In this embodiment, the communication connection between the device terminal and the control terminal is achieved through direct communication technology and the creation of a server management class. In this way, a connection can be quickly established between the device terminal and the control terminal without the need to pre-configure the network environment, thereby improving the user experience.

[0040] In one embodiment, a method for receiving a voice control instruction includes: calling a client processing class to create a client processing instance, wherein the inner class of the server management class includes the client processing class, and a first monitoring method is defined in the client processing class; through the client processing instance, a socket instance is called back; using the socket instance, an input stream acquisition method is called to obtain an input stream; and the first monitoring method is called to obtain the voice control instruction through the input stream.

[0041] In this embodiment, the server management class ServerManager also includes an inner class, the client processing class ClientHandler, which is used to handle interactions with the control terminal. The client processing class ClientHandler defines a method ①, the first listening method startListening(), which is used to listen for data transmitted from the device terminal.

[0042] In addition, the connection acceptance method serverSocket.accept() is a blocking method, which will return a ServerSocket instance when the control terminal is connected.

[0043] In this embodiment, after returning the ServerSocket instance, the client processing class ClientHandler is called to create a client processing instance, that is, the ClientHandler instance, which is used as the member variable ③ClientHandler instance of the server management class ServerManager, and through the ClientHandler instance, a callback is made to the ServerSocket instance. The input stream acquisition method socket.getInputStream() (provided by the operating system application programming interface) is called using the ServerSocket instance to obtain the input stream inputStream. The input stream inputStream can obtain data transmitted by the device terminal. At the same time, it is used as the member variable ①input stream inputStream of the client processing class ClientHandler. Finally, the first listening method startListening() is called to obtain the voice control instruction through the input stream inputStream.

[0044] In this embodiment, the client processing class is called to create a client processing instance, and based on the client processing instance, the socket instance is called back and the first monitoring method is called to monitor and obtain the voice control instructions through the internal class and callback mechanism. In this way, it can be ensured that the voice control instructions are received and processed in a timely manner.

[0045] In a possible embodiment, the client processing class ClientHandler further includes a member variable ② boolean variable isListening, that is, a Boolean variable listening state isListening, which is used to mark whether to continue to listen to the control terminal to send data.

[0046] As a possible embodiment, the startListening() method of the client processing class ClientHandler will determine whether to continue listening for the control terminal to send data based on the listening status isListening. If it is determined based on the listening status isListening that it is necessary to continue listening for the control terminal to send data, the input stream reading method inputStream.readLine() (natively provided by the operating system) is called to continuously read the data in the input stream inputStream. The inputStream.readLine() method is a blocking method and will only continue to execute when the control terminal sends data. Its return value is the data sent by the control terminal, and the read data is of string type. If it is determined based on the listening status isListening that it is not necessary to continue listening for the control terminal to send data, the client processing class ClientHandler will not perform any processing.

[0047] Step S320: Determine the target request type of the voice control instruction according to the request type identifier and a preset first mapping relationship, where the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type.

[0048] The first mapping relationship is pre-configured, and there may be a one-to-one correspondence between the request type identifier and the request type.

[0049] In addition, the request type refers to the type of user intention represented by the voice control command.

[0050] Step S330: determining a target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, where the second mapping relationship is a one-to-one correspondence between the request type and the processing class.

[0051] The second preset relationship is pre-configured, and there may be a one-to-one correspondence between the request type and the processing class.

[0052] In addition, the processing class refers to the implementation of the specific operation logic of the voice control instructions.

[0053] In one embodiment, the target request type is one of a page operation request, an application start request, and a semantic search request; the target processing class is one of a page information processing class, an application start processing class, and a semantic search processing class.

[0054] For example, if the target request type is a page operation request, the target processing class is a page information processing class; if the target request type is an application opening request, the target processing class is an application opening processing class; if the target request type is a semantic search request, the target processing class is a semantic search processing class.

[0055] For example, if the request type identifier is a page operation request identifier, the target request type is a page operation request, and the corresponding target processing class is the page information processing class NodeInfoHolder; if the request type identifier is an application open request identifier, the target request type is an application open request, and the corresponding target processing class is the application open processing class to OpenAppManager; if the request type identifier is a semantic search request identifier, the target request type is a semantic search request, and the corresponding target processing class is the semantic search processing class SchemaHanlder.

[0056] In this embodiment, in order to improve the convenience of voice control of smart electronic devices, that is, users can input various semantic information to control device operations, the user intentions corresponding to the user's voice information are classified, that is, based on the logic of the device terminal executing the voice control instructions, the request types corresponding to the voice control instructions are divided into page operation requests, application opening requests and semantic search requests. Among them, the page operation request is the operation request of the page root element control, the application opening request is the operation request to open the application, and the semantic search request is the operation request other than the operation request of the page root element control and the operation request to open the application, which is based on semantic analysis to obtain and execute device activities.

[0057] Exemplarily, page operation requests are applicable to scenarios where the device interface is visible and can be spoken. For example, the user sees the text "Start playing", "OK", etc. on the device interface. The user can input voice based on the page root element, and the control terminal transmits voice control instructions of the page operation request type; application opening request is applicable to the opening of application. For example, the user inputs voice "Open XX music software", and the control terminal transmits voice control instructions of the application opening request type; semantic search request is applicable to device operations other than the above two requests, that is, the voice input by the user is what the user says casually, which is not the root element of the page nor the opening of an application. For example, the user inputs voice "I want to listen to the song of XX (singer's name)", and the control terminal transmits voice control instructions of the semantic search request type.

[0058] For example, for the three types of voice control commands mentioned above, the "action" for page operation requests and application launch requests is fixed, such as "word" and "openApp," respectively. However, the "action" in user utterances can vary, such as "music" (I want to listen to xx's song) and "sms" (Send a text message to xx). Therefore, the "action" of semantic search requests is defined as "semantic." After receiving a voice control command from the control terminal and obtaining the request type identifier, the control terminal first determines whether the "action" is "word" or "openApp," and then confirms whether it is "semantic."

[0059] In this embodiment, different processing classes are configured according to voice control instructions of different request types. On the one hand, it not only meets various voice control scenarios of various smart electronic devices and improves the convenience of voice control of smart electronic devices, but also on the other hand, each processing class only needs to focus on the request type it is responsible for. In this way, the response speed of electronic devices to voice control instructions is improved, which can ensure that users receive accurate and timely feedback, thereby improving the overall user experience. In addition, the maintainability of the processing class is enhanced, and it is easier to locate the source of the error and perform targeted repairs when problems occur. Moreover, when it is necessary to add voice control instructions of new request types or modify the processing logic of existing instructions, it is only necessary to update or add the corresponding processing class, and its scalability is enhanced.

[0060] Step S340: calling the target processing class to operate the electronic device according to the request data.

[0061] In this embodiment, after the target processing class corresponding to the voice control instruction is determined, the request data is distributed to the target processing class, and the target processing class operates the electronic device according to the request data to complete the response to the voice control instruction.

[0062] In one embodiment, a target processing class is called to operate an electronic device according to the requested data, including: if the target request type is a page operation request, a page information processing class is obtained, the page information processing class includes an accessibility service class and a page root element processing class, the accessibility service class defines a root element acquisition method and implements a page root element listening interface; the page root element processing class is called to create a root element listening instance, and the root element acquisition method is called to obtain the page root element, and the page root element is called back and assigned to the root element listening instance; the electronic device is operated according to the assigned root element listening instance and the requested data.

[0063] In this embodiment, if the target request type is a page operation request (i.e., action=word), the execution logic of the request data in the voice control instruction is: pass the request data to the page information processing class NodeInfoHolder, operate the electronic device, that is, perform the page operation of the electronic device.

[0064] In this embodiment, the page information processing class NodeInfoHolder includes the accessibility service class MyAccessibilityService and the page root element processing class NodeInfoHandler. The accessibility service class MyAccessibilityService defines a root element acquisition method getRootInActiveWindow() and implements the page root element listening interface INodeInfoListener.

[0065] For example, the page root element listening interface INodeInfoListener defines a page root element obtaining method getNodeInfo(), which establishes a bridge between the accessibility service class MyAccessibilityService and the page root element processing class NodeInfoHandler.

[0066] In this embodiment, the accessibility service class MyAccessibilityService inherits from AccessibilityService (the accessibility service provided by the operating system). After guiding the user to enable the accessibility service function through the standard method provided by the operating system, the page root element processing class NodeInfoHandler creates a root element listening instance. The accessibility service class MyAccessibilityService obtains the current page root element through its internal getRootInActiveWindow() method. At the same time, the accessibility service class MyAccessibilityService implements the page root element listening interface INodeInfoListener. This interface is responsible for calling back the page root element obtained by the accessibility service class MyAccessibilityService to the page root element processing class NodeInfoHandler to implement the assignment of the root element listening instance. At this time, after the root element listening instance is assigned, the page root element processing class NodeInfoHandler can obtain the page root element. At the same time, this class can also obtain the request data, and can implement page operation logic based on the page root element and elements in the request data.

[0067] In this way, the introduction of accessibility service class, page root element processing class and page root element monitoring interface provides accessibility service function support for page operations. Through effective management of page root elements and monitoring of request data, efficient response to voice control instructions of page operation requests is achieved, thereby improving the overall user experience.

[0068] In one embodiment, an electronic device is operated based on an assigned root element monitoring instance and requested data, including: calling a find element method in a page root element processing class, traversing the page root element and the child elements corresponding to each page root element in the find element method; if there is a target element that matches the requested data, triggering a click event of the target element to operate the electronic device.

[0069] Illustratively, the element search method includes a public method findTextInNode(string:String,index:Int) and a private method findTextInNode(nodeInfo:AccessibilityNodeInfo).

[0070] Exemplarily, traversing the page root element and the sub-elements corresponding to each page root element in the element search method includes: calling a public method, obtaining the request data and the page root element in the root element listening instance, and passing it to a private method; calling a private method, traversing the page root element and the sub-elements corresponding to each page root element.

[0071] In this embodiment, the page root element processing class NodeInfoHandler defines a public method findTextInNode(string:String,index:Int) and a private method findTextInNode(nodeInfo:AccessibilityNodeInfo). The public method findTextInNode(string:String) is called when request data is distributed to obtain the request data. Its parameter string is the keyword said by the user, such as "OK". Then, within the public method, the page root element is obtained through the iNodeInfoListener instance and passed to the private method findTextInNode(nodeInfo:AccessibilityNodeInfo). The private method recursively traverses the root element and its child elements. If a target element matching the request data is found, the click event of the target element is triggered to perform the page operation.

[0072] Exemplarily, triggering a click event of a target element means executing a click method provided by the operating system to realize a click on the target element.

[0073] In this embodiment, a search element method is defined in the page root element processing class, and the root element and its sub-elements are traversed in combination with the request data, thereby achieving accurate positioning of the target element corresponding to the request data and ensuring the reliability of page operations.

[0074] In one embodiment, a method for determining a target element includes: comparing an index parameter with a preset parameter threshold, where the index parameter is carried in a voice control instruction and is an identifier that uniquely identifies the target element indicated by the request data; if the index parameter is greater than the parameter threshold, then in the process of traversing the root element and child elements of the page, the number of times the element that matches the request data is traversed is recorded, and if the number matches the index parameter, the element corresponding to the number is determined as the target element.

[0075] Exemplarily, if the index parameter is less than or equal to the parameter threshold, the element that is first traversed and matches the requested data is determined as the target element.

[0076] In this embodiment, considering that multiple elements in the current page root element and its child elements have the same keyword, for example, including multiple "OK", when the control terminal encapsulates the voice control instruction, if the request type is a page operation request, an index parameter field is set when encapsulating the voice control instruction. The index parameter field describes the index parameter. Therefore, the voice control instruction received by the electronic device also carries the index parameter. The index parameter is used to identify the target element among the multiple elements when multiple elements have the same keyword and the keyword of the element represented by the request data happens to be this word.

[0077] Exemplarily, "index":"..." is used as the request type identification field, where "..." is a numerical value.

[0078] For example, when there are three elements whose keywords are all "OK", these three OKs correspond to different indexes. The index corresponding to the first OK is 0, the index corresponding to the second OK is 1, and the index corresponding to the third OK is 2. If the user wants to click the second OK, then index=1.

[0079] In this embodiment, before traversing the root element and its child elements, the index parameter is compared with a preset parameter threshold. If the index parameter is less than or equal to the parameter threshold, the element that is traversed for the first time and is consistent with the requested data is determined as the target element during the traversal. If the index parameter is greater than the parameter threshold, the number of times the element that is consistent with the requested data is traversed is recorded during the traversal, and if the number is consistent with the index parameter, the element corresponding to the number is determined as the target element.

[0080] For example, the parameter threshold is set to 0. When the value of index is less than or equal to 0 (less than for fault tolerance), the click event is triggered when the first element that matches the requested data is traversed. When index is greater than 0, the number of occurrences of elements that match the requested data is recorded, and the click event is triggered when the number of occurrences matches the value of index. If the recorded number of occurrences still does not match the value of index after traversing all root elements and their child elements, it is considered a failure.

[0081] For example, if the keyword in the request data is "OK", when index=0, a click event is triggered when the first "OK" is traversed; when index=3, a click event is triggered when the fourth "OK" is traversed.

[0082] As a possible embodiment, the judgment rule of whether the number of times is consistent with the index parameter can be set according to the needs. For example, the above-mentioned "consistent" judgment rule is: number of times = index parameter + 1. Of course, the "consistent" judgment rule can also be: number of times = index parameter + 2, number of times = index parameter or number of times = index parameter - 1, etc. The embodiment of the present application does not make specific restrictions.

[0083] In this embodiment, by introducing index parameters and threshold mechanisms, when there are multiple elements on a page with the same keyword, the target element can be determined more accurately, avoiding the possibility of misoperation and further ensuring the reliability of page operation.

[0084] See Figure 4 , Figure 4 This is a page information processing class diagram shown in an exemplary embodiment of the present application. Figure 4 As shown, the page information processing class includes an accessibility service class and a page root element processing class. The accessibility service class inherits from the accessibility service provided by the operating system. The accessibility service defines a method for obtaining the current page root element. Therefore, the accessibility service class inherits this method. In addition, the implementation of obtaining the page root element in the accessibility service class is implemented by defining a page root element listening interface. When calling the page root element processing class, a root element listening instance can be created and the page root element can be passed through the page root element listening interface to assign the root element listening instance value, and then the keyword in the request data can be obtained to perform element traversal.

[0085] In a possible embodiment, after triggering a click event of the target element and executing the page operation, it also includes: obtaining the page operation result; if the operation is successful, assembling the information of the successful operation and returning it to the control terminal; if the operation fails, assembling the information of the failed operation and returning it to the control terminal.

[0086] Illustratively, reasons for the operation failure include: failure due to failure to traverse an element that matches the requested data, and failure due to failure to implement a click event.

[0087] For example, when the operation is successful, the returned message is: {"responseId":"message":"Operation successful"}; when the operation fails: {"responseId":"message":"Operation failed":"Reason for failure"}.

[0088] In one embodiment, a target processing class is called to perform an operation on an electronic device according to the request data, including: if the target request type is an application opening request, an application opening processing class is obtained; the application opening processing class is called to obtain an application information list, and the application names in the application information list are traversed, the application information list includes an application name and an application package name; if there is a target application name that matches the request data, the target application package name is determined from the application information list according to the target application name, and an application start event of the target application corresponding to the target application package name is triggered to operate the electronic device.

[0089] In this embodiment, if the target request type is an application opening request (i.e., action=openApp), the execution logic of the request data in the voice control instruction is: pass the request data to the application opening processing class OpenAppManager, operate the electronic device, that is, execute the application opening operation.

[0090] In this embodiment, the application opening processing class OpenAppManager obtains the application information list installed on the electronic device, which includes the application name and the application package name (the application name is used to match the request data, and the application package name is used to match the target application), and then traverses the application names in the application information list. If there is a target application name that matches the request data, the target application package name is determined, and the application start event of the target application corresponding to the target application package name is triggered to perform the application opening operation.

[0091] For example, the application opening processing class OpenAppManager can obtain all application information installed on the electronic device through the application information obtaining method PackageManager() provided by the operating system, and generate an application information list.

[0092] Exemplarily, triggering an application start event of a target application corresponding to the target application package name means executing a start method startActivity() provided by the operating system to start the target application.

[0093] In this embodiment, an application start processing class is introduced to achieve accurate matching of the application package name of the target application to start the correct application, avoid erroneous operation of application start, and ensure the reliability of the application opening operation.

[0094] See Figure 5 , Figure 5 This is an example of an application start-up process diagram shown in an exemplary embodiment of the present application. Figure 5 As shown, the application opening processing class implements the opening operation of the target application through the application package name.

[0095] In a possible embodiment, after triggering the application startup event of the target application corresponding to the target application package name and executing the application opening operation, it also includes: obtaining the application opening operation result; if the operation is successful, the information of the successful assembly operation is returned to the control terminal; if the operation fails, the information of the failed assembly operation is returned to the control terminal.

[0096] Illustratively, reasons for the operation failure include: failure due to failure to traverse an application name that matches the request data, and failure due to failure to implement an application startup event.

[0097] For example, when the operation is successful, the returned message is: {"responseId":"message":"Operation successful"}; when the operation fails: {"responseId":"message":"Operation failed":"Reason for failure"}.

[0098] In one embodiment, a target processing class is called to operate an electronic device according to the request data, including: if the target request type is a semantic search request, obtaining a semantic search processing class; calling the semantic search processing class to obtain a data parsing method and an activity start method; using the data parsing method to parse the request data to obtain an activity instance, which includes a target application package name and an activity name; passing the activity instance to the activity start method to trigger an activity start event to operate the electronic device.

[0099] In this embodiment, if the target request type is an application start request (i.e., action=semantic), the execution logic of the request data in the voice control instruction is: pass the request data to the semantic search processing class SchemaHanlder, operate the electronic device, that is, perform the activity start operation.

[0100] In this embodiment, the data parsing method Uri.parse(schema) and the activity start method startActivity() provided by the operating system are obtained through the semantic search processing class SchemaHanlder. The request data is first parsed using the Uri.parse(schema) method to obtain an activity instance, which includes the target application package name and the activity name. The activity instance is then passed to startActivity() to trigger an activity start event and execute the activity start operation.

[0101] Exemplarily, the request data is parsed using the Uri.parse(schema) method to obtain a uniform resource identifier instance, ie, a Uri instance, and then the Uri instance is passed to an intent instance, ie, an Intent instance, which is the final activity instance.

[0102] Exemplarily, passing the activity instance to the activity start method triggers the activity start event and executes the activity start operation, that is, the activity start method executes the activity start operation according to the target application package name and activity name in the activity instance.

[0103] In this embodiment, by introducing a semantic search processing class, support is provided for more complex voice input, voice control of intelligent electronic devices under complex voice input is realized, and user experience is improved.

[0104] See Figure 6 , Figure 6 This is a semantic search processing class diagram shown in an exemplary embodiment of the present application. Figure 6 As shown, the semantic search processing class implements activity processing by requesting data.

[0105] In a possible embodiment, after passing the activity instance to the activity start method to trigger the activity start event and executing the activity start operation, it also includes: obtaining the activity start operation result; if the operation is successful, assembling the information of the successful operation and returning it to the control terminal; if the operation fails, assembling the information of the failed operation and returning it to the control terminal.

[0106] Illustratively, reasons for the operation failure include: failure due to failure to traverse an application package name or activity name that matches the request data, and failure due to failure to implement an activity start event.

[0107] For example, when the operation is successful, the returned message is: {"responseId":"message":"Operation successful"}; when the operation fails: {"responseId":"message":"Operation failed":"Reason for failure"}.

[0108] In one embodiment, after operating the electronic device according to the requested data, it also includes: using the socket instance to call the output stream acquisition method to obtain the output stream; calling the server data sending method, executing the first data transmission method defined in the client processing class, writing the device operation result to the output stream and sending it to the control terminal, and the server management class also defines a server data sending method.

[0109] In this embodiment, the client processing class ClientHandler also defines method ② the first data transmission method sendMessage(message:String), which is used to send data to the control terminal; the server management class ServerManager also defines method ② the server data sending method ServerManager.sendMessage(message:String), which is used to call the sendMessage(message:String) method of the client processing class ClientHandler to directly send data to the device terminal.

[0110] In this embodiment, the output stream acquisition method socket.getOutputStream() (provided by the operating system application programming interface) is called using the ServerSocket instance to obtain the output stream outputStream. The input stream outputStream can obtain the data that needs to be transmitted to the device terminal. At the same time, it is used as the member variable ② output stream outputStream of the client processing class ClientHandler, and then the server data sending method ServerManager.sendMessage(message:String) is called to execute the first data transmission method sendMessage(message:String) defined in the client processing class ClientHandler, write the device operation result into the output stream outputStream and send it to the control terminal, so as to realize the feedback of the device operation result.

[0111] Exemplarily, executing the first data transmission method defined in the client processing class, writing the device operation result into the output stream and sending it to the control terminal includes: using the first data transmission method sendMessage (message:String), calling the data writing method outputStream.println(message) and the data sending method outputStream.flush() provided by the operating system (provided by the operating system); executing the outputStream.println(message) method, writing the device operation result into the output stream outputStream; executing the outputStream.flush() method, sending the output stream outputStream of the written device operation result to the control terminal.

[0112] In this embodiment, instant feedback of device operation results is achieved through the client processing class, thereby improving the user's interactive experience.

[0113] In a possible embodiment, the server management class ServerManager further defines a method ③ connection closing method closeConnection(), and the client processing class ClientHandler further defines a method ③ stop listening method stopListening(), which is used to send data to the control terminal.

[0114] As a possible embodiment, the closeConnection() method of the server management class ServerManager calls the stopListening() method of the client processing class ClientHandler to determine whether to continue listening to the data sent by the control terminal. The implementation method is to determine whether to continue listening to the data sent by the control terminal based on the listening status isListening. If the listening status isListening is set to false, the listening is stopped, and the closeConnection() method is called to close the connection with the control terminal. The implementation method is to call the socket closing method socket.close() inside the closeConnection() method to close the connection. After the connection is closed, the thread pool executorService is closed.

[0115] See Figure 7 , Figure 7 This is a device terminal class diagram shown in an exemplary embodiment of the present application. Figure 7As shown, at the device terminal, the direct connection management class includes an initialization method for instantiating the direct connection management class, a direct connection group creation method for creating a direct connection group and passing in a channel instance, and a group removal method for removing the direct connection group; the channel class includes a closing method for closing the channel after closing the connection; the server management class includes member variables server socket, thread pool object and client processing instance, including server startup method, server data sending method and connection closing method; the client processing class includes member variables input stream, output stream and listening status, including listening method, data transmission method and stop listening method.

[0116] In one embodiment, before receiving the voice control instruction from the control terminal, it also includes: detecting whether the electronic device has a voice capture and recognition function; if the electronic device does not have the voice capture and recognition function, sending a voice control instruction through the control terminal to control the electronic device; if the electronic device has a voice capture and recognition function, detecting whether the voice capture and recognition function has a fault, and if the voice capture and recognition function has a fault, sending a voice control instruction through the control terminal to control the electronic device.

[0117] In this embodiment, taking into account the problem that the user experience is affected by the fact that the electronic device does not have the voice capture and recognition function or the voice capture and recognition function malfunctions, whether the electronic device has the voice capture and recognition function or whether the voice capture and recognition function malfunctions is detected in advance, and when the voice capture and recognition function is not available or the voice capture and recognition function malfunctions, the solution for implementing device voice control based on the control terminal is adaptively started. In this way, the redundancy of the voice control of the electronic device is increased through the adaptive switching mechanism, thereby improving the user experience.

[0118] Exemplarily, detecting whether there is a fault in the voice capture and recognition function includes: obtaining the operating status of the microphone and the voice assistant, the operating status including normal operation and abnormal operation; if the operating status of the microphone and / or the operating status of the voice assistant is abnormal operation, a prompt message is generated and popped up, the prompt message is whether to enable the device control scheme based on the control terminal, and the electronic device control scheme is enabled under the condition that the enable instruction is received; if the operating status of the microphone and the operating status of the voice assistant are normal operation, the microphone's reception status is determined according to the voice recognition accuracy information and / or voice recognition efficiency information of the voice assistant, the reception status including normal reception and abnormal reception, and under the condition that the microphone's reception status is abnormal reception, a prompt message is generated and popped up, and the electronic device control scheme is started under the condition that the enable instruction is received.

[0119] Among them, the operating status of the microphone and voice assistant indicates whether they are available, that is, normal operation indicates available, and abnormal operation indicates unavailable; the microphone's reception status indicates whether the microphone receives sound accurately, that is, normal reception indicates accurate, and abnormal reception indicates inaccurate, and the microphone performance is impaired.

[0120] In this embodiment, when an electronic device is configured with a microphone, the operating status and reception status of the microphone are judged in advance, taking into account the problem that the user experience may be affected by situations such as microphone failure or poor performance. When the operating status of the microphone is abnormal or the reception status is abnormal, the solution for implementing device voice control based on the control terminal is adaptively started. In this way, the redundancy of the electronic device's voice control is increased through the adaptive switching mechanism. When the microphone fails or performs poorly, the control terminal is used to implement the electronic device's voice control, ensuring the stability of the device's voice control function and improving the user experience.

[0121] For example, the operating status of the microphone is determined by the hardware operating status monitoring system. The specific determination method can be implemented based on existing methods, and the operating status of the voice assistant can also be implemented based on existing methods, which will not be described in detail in this embodiment of the application.

[0122] Exemplarily, the reception status is determined based on the voice recognition accuracy information and / or voice recognition efficiency information of the voice assistant, the voice recognition accuracy information includes the voice recognition accuracy, and the voice recognition efficiency information includes the voice recognition efficiency value. When determined based on the voice recognition accuracy information, if the voice recognition accuracy value is greater than or equal to the preset accuracy value, the reception is determined to be normal; otherwise, it is abnormal. When determined based on the voice recognition efficiency information, if the voice recognition efficiency value is greater than or equal to the preset efficiency value, the reception is determined to be normal; otherwise, it is abnormal. When determined based on the voice recognition accuracy information and the voice recognition efficiency information, the voice recognition accuracy value, the voice recognition efficiency value and their respective preset weights are calculated. If the calculation result is greater than or equal to the preset value, the reception is determined to be normal; otherwise, it is abnormal.

[0123] Exemplarily, the sound reception status is determined based on the voice recognition accuracy value and / or voice recognition efficiency value of the voice assistant for historical voice commands, wherein historical voice commands refer to voice control commands directly captured by the microphone configured on the device during historical control, the voice recognition accuracy value is the average of the most recent recognition accuracy in the voice assistant's historical recognition, and the voice recognition efficiency value is the average of the most recent recognition efficiency in the voice assistant's historical recognition. During the historical recognition process, the accuracy and efficiency of each recognition will be recorded. In addition, the specific confirmation method for the accuracy and efficiency of each recognition can be implemented based on existing methods, which will not be elaborated in the embodiments of this application.

[0124] In the above-mentioned electronic device control method, the device terminal establishes a communication connection with the control terminal by creating a direct connection group to receive a voice control instruction from the control terminal. The voice control instruction carries a request type identifier and request data. Then, based on the request type identifier and a preset first mapping relationship, the target request type of the voice control instruction is determined. The first mapping relationship is a one-to-one correspondence between the request type identifier and the request type. Then, based on the target request type and a preset second mapping relationship, the target processing class of the voice control instruction is determined. The second mapping relationship is a one-to-one correspondence between the request type and the processing class. Finally, the target processing class is called to operate the electronic device according to the request data. The control terminal captures the user voice and controls the electronic device by receiving the voice control instruction transmitted by the control terminal. Moreover, for different voice control requests, the device terminal is configured with different processing classes, which can meet the precise execution of different voice control requests. In this way, through the microphone-free voice adaptation solution, voice control of electronic devices is realized without relying on a microphone, which reduces costs and improves user experience.

[0125] Moreover, if electronic devices do not need to be equipped with microphones, it not only reduces costs but also enables voice control of electronic devices. For electronic devices equipped with microphones, a backup solution for voice control is provided, which expands the control range of electronic devices and makes it easier for users who are temporarily unable to interact with electronic devices at close range to use and operate electronic devices.

[0126] See Figure 8 , Figure 8 This is a flow chart of another electronic device control method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 or Figure 2 The implementation environment shown is shown. It should be understood that the method can also be applied to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0127] In addition, the electronic device control method can be applied to control terminals under various operating systems, including but not limited to control terminals under the Android operating system.

[0128] like Figure 8 As shown, in an exemplary embodiment, the electronic device control method includes at least steps S810 to S840, which are described in detail as follows: Step S810: If the communication connection with the device terminal is successfully established, the user audio data is captured and converted into text data. The communication connection is established through a direct connection group created by the device terminal. The device terminal is independent of the electronic device or integrated into the electronic device.

[0129] For example, by integrating various third-party toolkits such as iFlytek, Google and Baidu's speech recognition services, user audio data is captured according to the third-party toolkit and converted into text data.

[0130] In one embodiment, a method for establishing a communication connection includes: obtaining a second direct connection management class and creating a broadcast receiver, wherein the second direct connection management class defines a group owner search method and a connection method; calling the group owner search method to search for group owners within a preset distance range, and using the broadcast receiver to listen and call back the group owner information; if a connection event is triggered, creating a direct connection instance based on the group owner information of the target group owner; calling the connection method, accessing the direct connection group based on the direct connection instance, and establishing a communication connection.

[0131] The second direct connection management class is pre-configured on the control terminal.

[0132] Exemplarily, the first direct connection management class is the Wi-Fi direct connection management class WifiP2pManager.

[0133] In this embodiment, the second direct connection management class, WifiP2pManager, is obtained using the direct connection management class acquisition method provided by the operating system. After initialization, the initialize() method of the second direct connection management class, WifiP2pManager, is called. A channel, Channel, is created to obtain a channel instance, i.e., a Channel instance. A broadcast receiver, ClientBroadcastReceiver, is created to listen for and call back group owner information. The second direct connection management class, WifiP2pManager, defines the group owner search method, wifiP2pManager.discoverPeers(), and the connection method, wifiP2pManager.connect().

[0134] In this embodiment, after setting up the monitoring, wifiP2pManager.discoverPeers() is called to search for the group owner (device terminal) (the device terminal can be searched by the control terminal only after the direct connection group is successfully created). During the search, the broadcast receiver ClientBroadcastReceiver is used to monitor and call back the group owner information (such as Internet Protocol address, name, etc.) to obtain the group owner list. When the device terminal is searched and connected, the Channel instance is passed in, and a direct connection instance, that is, a WifiP2pConfig instance, is created based on the Internet Protocol address of the device terminal. Finally, the WifiP2pConfig instance is passed to the wifiP2pManager.connect() method to establish a direct connection group and establish a communication connection.

[0135] Exemplarily, the broadcast receiver ClientBroadcastReceiver may also monitor changes in group owners and connection status with each group owner.

[0136] For example, when the connection is successful, the device terminal host information InetAddress will be called back through the broadcast receiver ClientBroadcastReceiver monitoring.

[0137] In this embodiment, the communication connection between the control terminal and the device terminal is achieved through direct communication technology and the creation of broadcast receivers. In this way, a connection can be quickly established between the control terminal and the device terminal without the need to pre-configure the network environment, thereby improving the user experience.

[0138] In one embodiment, after the communication connection is successfully established, it also includes: calling back the host information of the device terminal; creating a socket management class, in which a second monitoring method and a second data transmission method are defined, the second monitoring method is used to monitor the device operation results from the device terminal according to the host information, and the second data transmission method is used to send voice control instructions to the device terminal according to the host information.

[0139] In this embodiment, after the communication connection is successfully established, it is necessary to implement interaction with the device terminal, that is, sending data (mainly voice control instructions) and receiving data (mainly device operation results). This is achieved by creating a socket management class SocketManager, which is responsible for managing the socket of the control terminal and the host information InetAddress of the device terminal that needs to be called back.

[0140] Among them, the socket management class SocketManager defines method ① the second listening method listenForMessages() and method ② the second data transmission method sendMessage(message:String). By calling the listenForMessages() method, the device operation results from the device terminal are listened to according to the host information, and by calling the sendMessage(message:String) method, voice control instructions are sent to the device terminal according to the host information.

[0141] Illustratively, the sending of the sound control instruction is achieved by calling the data writing method outputStream.println(message) and the data sending method outputStream.flush() (provided by the operating system) through the sendMessage(message:String) method.

[0142] In this embodiment, a socket management class is created, and interaction with the device terminal is achieved based on the second monitoring method and the second data transmission method of the socket management class, that is, sending voice control instructions and receiving device operation results. In this way, the instant sending and receiving of voice control instructions and device operation results can be ensured.

[0143] In a possible embodiment, the socket management class SocketManager includes member variables ① socket Socket, which is responsible for transmitting data with the device terminal and is initialized by the host information InetAddress of the device terminal and the port number consistent with the device terminal; member variable ② input stream inputStream, which is obtained by the input stream acquisition method socket.getInputStream() provided by the native operating system, and the input stream inputStream can obtain the data transmitted from the device terminal; member variable ③ output stream outputStream, which is obtained by the output stream acquisition method socket.getOutputStream() provided by the native operating system, and the output stream outputStream can transmit data to the device terminal.

[0144] For example, the listenForMessages() method reads the data transmitted by the device terminal according to whether the socket is connected. If it is still connected, the input stream acquisition method inputStream.readLine() (natively provided by the operating system) is called to continuously read the data in the input stream inputStream. If the socket is disconnected, the control terminal does not perform any processing.

[0145] In a possible embodiment, the socket management class SocketManager also defines method ③ initialization method init(), which is responsible for initializing member variables socket, input stream and output stream; method ④ connection closing method closeConnection(), which is used to disconnect the socket connection, thereby disconnecting from the device terminal.

[0146] In a possible embodiment, after receiving the device operation result, the device operation result is announced by voice, including a successful operation, or a failed operation and the reason for the failure.

[0147] See Figure 9 , Figure 9 This is a control terminal class diagram shown in an exemplary embodiment of the present application. Figure 9As shown, in the control terminal, the direct connection management class includes an initialization method for instantiating the direct connection management class, a connection method for passing in a channel instance when connecting, and a group owner search method for searching for devices; the channel class includes a close method for closing the channel after closing the connection; the broadcast receiver includes member variables of the direct connection management class and the device listener, as well as a group owner information callback function; the socket management class includes member variables of the socket, input stream and output stream, and includes an initialization method, a listening method, a data transmission method and a connection closing method.

[0148] Step S820: Parse the text data into formatted data, where the formatted data includes a request type identifier and request data. The request type identifier is determined based on the request type of the text data and a first mapping relationship, where the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type.

[0149] The first mapping relationship is a preset configuration, and the request type identifier and the request type may be in a one-to-one relationship.

[0150] In addition, the request type refers to the type of user intention represented by the voice control command.

[0151] In one embodiment, the request type is one of a page operation request, an application start request, and a semantic search request; if the request type is a page operation request, the formatted data also includes an index parameter, which is an identifier that uniquely identifies the target element pointed to by the request data.

[0152] In this embodiment, when the request type is a page operation request, considering that the keywords of multiple elements in the current page root element and its child elements are the same, an index parameter is introduced. In this way, when the keywords of multiple elements are the same, it can identify the target element corresponding to the request data and ensure the reliability of subsequent device terminal execution.

[0153] Exemplarily, the text data is parsed into formatted data according to a preset parsing format. If the request type is a page operation request, the preset parsing format includes a request type identification field, a request data field, and an index parameter field; if the request type is an application launch request or a semantic search request, the preset parsing format includes a request type identification field and a request data field.

[0154] Exemplarily, the preset parsing format is a Json string.

[0155] As a possible implementation, three types of semantics are supported: ① Keywords, suitable for visible-speakable scenarios, such as text the user sees on the device interface, for example, "Start playing." A third party will parse the user's semantics and return them as a JSON string, {"action":"word","data":"Start playing","index":0}; ② Open an application, a fixed phrase, for example, "Open a certain music app." Based on "open," the string is parsed and returned as a JSON string, {"action":"opeanApp","data":"a certain music app"}; ③ User-generated phrases, such as "I want to listen to a song by (singer's name)," are parsed and returned as a JSON string, {"action":"semantic,"data":"music: / / artist?name=a certain (singer's name)"}. The formatted data obtained after parsing is then further encapsulated into a voice control command.

[0156] Step S830: Encapsulate the formatted data into a voice control instruction, where the voice control instruction carries a request type identifier and request data.

[0157] Exemplarily, the formatted data is encapsulated into a voice control command according to a preset encapsulation format. If the request type is a page operation request, the preset encapsulation format includes a request type identification field, a request data field, and an index parameter field. If the request type is an application launch request or a semantic search request, the preset parsing format includes a request type identification field and a request data field.

[0158] Exemplarily, no matter whether the request type is a page operation request, an application start request, or a semantic search request, the preset encapsulation format further includes a communication identification field.

[0159] In this way, a communication between the device terminal and the control terminal can be identified, making it easier to track the communication connection process between the two and the flow path of the interactive information, monitor the entire life cycle of the communication, and facilitate problem troubleshooting and log analysis.

[0160] For example, if the request type is a page operation request, taking voice control clicking on the page element "OK" as an example, the voice control instruction generated by the package is as follows: { "requestId":"a1d16b58083347d19740198f43d7ee22", "action":"word", "data":"OK", "index":0 } "requestId":"a1d16b58083347d19740198f43d7ee22" is the communication identifier, which can be a 32-bit unique identifier generated by the Java UUID (Universally Unique Identifier) ​​tool class. "action":"word" is the request type identifier. "data":"OK" is the request data field, which is the element visible on the device terminal screen that the user requested. "index":0 is the index identifier field, and its value depends on the user's statement. For example, if the user says "second OK," the value is 1; if the user says "first OK," the value is 0. If the user does not specify the nth OK, the default is the first OK, and the default index is 0.

[0161] For example, if the request type is an application opening request, taking voice control to open a certain music software as an example, the encapsulated and generated voice control instruction is as follows: { "requestId":"de06b23e5ea9408785fe76b87d64e72", "action":"openApp", "data":"Music software" } Among them: "requestId":"de06b23e5ea9408785fe76b87d64e72" is the communication identification field, which can be a 32-bit unique identifier generated by the UUID tool class provided by Java; "action":"openApp" is the request type identification field; "data":"XX Music Software" is the request data field, that is, the name of the application the user wants to open.

[0162] For example, if the request type is a semantic search request, taking the example of listening to a song by (singer's name), the encapsulated and generated voice control command is as follows: { "requestId":6fc07c6c60704212bc1af04870130196", "action":"semantic", "data":"music: / / artist?name=XXX (singer's name)" } Among them: "requestId": 6fc07c6c60704212bc1af04870130196" is the request type identification field, which can be a 32-bit unique identifier generated by the UUID utility class provided by Java; "action": "semantic" is the request type identification field; "data": "music: / / artist?name=某某(歌手姓名)" is the request data field, that is, the device activity operation desired by the user.

[0163] In addition, the request data field in the format of "data": "music: / / artist?name=某某(歌手姓名)" is only illustrated by taking music as an example. In the actual implementation process, it can be adapted according to the URL Scheme (Uniform Resource Locator Scheme) supported by various applications in the market. There will be an explanation of the URL Scheme in each application developer center, so as to configure the request data field.

[0164] Step S840, send a voice control instruction to the device terminal so that the device terminal operates the electronic device according to the voice control instruction.

[0165] At this point, the control terminal completes the sending of the voice control instruction and performs voice control on the electronic device.

[0166] In the above electronic device control method, after the control terminal establishes a communication connection with the device terminal, it starts to capture user audio data and convert it into text data, then analyzes the text data into formatted data. The formatted data includes the request type identification and the request data. Then, the formatted data is encapsulated into a voice control instruction. The voice control instruction carries the request type identification and the request data. Finally, the voice control instruction is sent to the device terminal so that the device terminal operates the electronic device according to the voice control instruction, realizing the voice control of the electronic device. It provides support for different user intentions, that is, diverse user voices, generates voice control instructions of different request types, and is also convenient for the device terminal to classify and process, so as to meet the precise execution of different voice control requests. In this way, through the microphone-free voice adaptation scheme, the voice control of the electronic device is realized without relying on the microphone, which not only reduces the cost but also improves the user experience.

[0167] Please refer to Figure 10 , Figure 10 which is a flowchart of a specific electronic device control method shown in an exemplary embodiment of the present application. The specific electronic device control method includes the establishment of a communication connection between the device terminal and the control terminal, the transfer of data, and the response of the device terminal to the voice control instruction. As Figure 10As shown, taking the electronic device as a vehicle system and the device terminal as a vehicle-mounted terminal as an example, the specific electronic device control method is detailed as follows: The first step is to establish a communication connection. First, initialize the direct connection management class of the vehicle side and the control terminal respectively. In the vehicle side, after initialization, create a direct connection group through the direct connection management class and wait for the control terminal to connect. In the control terminal, after initialization, create a direct connection instance through the direct connection management class, create a broadcast receiver and initialize it; then the control terminal searches for the group owner through the direct connection management class, and calls back after searching the vehicle side. The broadcast receiver displays it to the user. The user clicks on the vehicle side device to connect, triggering the connection event of the control terminal and accessing the direct connection group of the vehicle side. If the connection is successful, the control terminal calls back the host information of the vehicle side, and the vehicle side waits for the control terminal to send information. At this point, the vehicle side and the control terminal complete the communication connection; Next comes data transmission. The control terminal captures the user's input audio data through the socket management class, converts it into text data, parses and encapsulates it, obtains the voice control command, and sends it to the vehicle end. The voice control command carries the request type identifier and request data. Next, the vehicle responds to the voice control command. The vehicle's service management class determines the request type based on the request type identifier and selects the corresponding processing class from the page operation processing class, application launch processing class, and semantic search processing class to respond to the request data and execute the vehicle operation. Finally, there is data transmission, that is, after the vehicle side completes the execution, it sends the vehicle computer operation result to the control terminal, and the control terminal receives the vehicle computer operation result and broadcasts it in voice.

[0168] See Figure 11 , Figure 11 This is a block diagram of an electronic device control system shown in an exemplary embodiment of the present application. The system can be applied to Figure 1 or Figure 2 The implementation environment shown is shown. It should be understood that the system can also be applied to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the system is applicable.

[0169] like Figure 11 As shown, in an exemplary embodiment, the electronic device control system 1100 includes at least a device terminal 1110 and a control terminal 1120, which are described in detail as follows: Device terminal 1110, used to create a direct connection group, the device terminal is independent of the electronic device or integrated into the electronic device; The control terminal 1120 is configured to establish a communication connection with the device terminal 1110 according to the direct connection group; if the communication connection is successfully established, capture user audio data and convert the user audio data into text data; parse the text data into formatted data, the formatted data including a request type identifier and request data, the request type identifier being determined based on a request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; encapsulate the formatted data into a voice control instruction, the voice control instruction carrying the request type identifier and the request data; and send the voice control instruction to the device terminal; The device terminal 1110 is also used to receive voice control instructions from the control terminal 1120; determine the target request type of the voice control instruction based on the request type identifier in the voice control instruction and the first mapping relationship; determine the target processing class of the voice control instruction based on the target request type and a preset second mapping relationship, the second mapping relationship being a one-to-one correspondence between the request type and the processing class; call the target processing class and operate the electronic device according to the request data in the voice control instruction.

[0170] It should be noted that the electronic device control system provided in the above embodiment and the electronic device control method applied to the device terminal and the control terminal provided in the above embodiment belong to the same concept, and the content of the operations performed by each module has been described in detail in the method embodiment and will not be repeated here.

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0173] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, any equivalent modifications or alterations accomplished by a person of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A method for controlling an electronic device, characterized in that: Applied to a device terminal, the device terminal is independent of the electronic device or integrated in the electronic device, the method comprises: Receiving a voice control instruction from a control terminal, the device terminal establishing a communication connection with the control terminal by creating a direct connection group, the voice control instruction carrying a request type identifier and request data; Determining a target request type of the voice control instruction according to the request type identifier and a preset first mapping relationship, wherein the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type; Determining a target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, wherein the second mapping relationship is a one-to-one correspondence between the request type and the processing class; The target processing class is called to operate the electronic device according to the request data.

2. The electronic device control method according to claim 1, characterized in that: The target request type is one of a page operation request, an application start request and a semantic search request; the target processing class is one of a page information processing class, an application start processing class and a semantic search processing class.

3. The electronic device control method according to claim 1, characterized in that: The calling of the target processing class and operating the electronic device according to the request data includes: If the target request type is a page operation request, a page information processing class is obtained, wherein the page information processing class includes an accessibility service class and a page root element processing class, wherein a root element obtaining method is defined in the accessibility service class and a page root element monitoring interface is implemented; Call the page root element processing class to create a root element listening instance, call the root element acquisition method to acquire the page root element, and call back the page root element and assign it to the root element listening instance; The electronic device is operated according to the assigned root element monitoring instance and the request data.

4. The electronic device control method according to claim 3, characterized in that: The step of operating the electronic device according to the assigned root element monitoring instance and the request data includes: Calling the search element method in the page root element processing class, and traversing the page root element and the sub-elements corresponding to each of the page root elements in the search element method; If there is a target element matching the request data, a click event of the target element is triggered to operate the electronic device.

5. The electronic device control method according to claim 4, characterized in that: The method for determining the target element includes: Comparing an index parameter with a preset parameter threshold, wherein the index parameter is carried in the voice control instruction, and the index parameter is an identifier that uniquely identifies a target element indicated by the request data; If the index parameter is greater than the parameter threshold, then in the process of traversing the page root element and the child elements, the number of times the element matching the request data is traversed is recorded, and when the number matches the index parameter, the element corresponding to the number is determined as the target element.

6. The electronic device control method according to claim 1, characterized in that: The calling of the target processing class and operating the electronic device according to the request data includes: If the target request type is a semantic search request, obtaining a semantic search processing class; Call the semantic search processing class to obtain the data parsing method and activity starting method; Parsing the request data using the data parsing method to obtain an activity instance, wherein the activity instance includes a target application package name and an activity name; Passing the activity instance to the activity start method triggers an activity start event to operate the electronic device.

7. The electronic device control method according to any one of claims 1 to 6, characterized in that: Before receiving the voice control instruction from the control terminal, the method further includes: Detecting whether the electronic device has a voice capture and recognition function; If the electronic device does not have the voice capture and recognition function, sending a voice control command through the control terminal to control the electronic device; If the electronic device has the voice capture and recognition function, it is detected whether the voice capture and recognition function has a fault, and if the voice capture and recognition function has a fault, a voice control instruction is sent through the control terminal to control the electronic device.

8. A method for controlling an electronic device, characterized in that: Applied to a control terminal, the method comprises: If the communication connection with the device terminal is successfully established, capturing user audio data and converting the user audio data into text data, wherein the communication connection is established through a direct connection group created by the device terminal, and the device terminal is independent of the electronic device or integrated in the electronic device; Parsing the text data into formatted data, the formatted data including a request type identifier and request data, the request type identifier being determined according to a request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; Encapsulating the formatted data into a voice control instruction, wherein the voice control instruction carries the request type identifier and the request data; The voice control instruction is sent to the device terminal so that the device terminal operates the electronic device according to the voice control instruction.

9. The electronic device control method according to claim 8, characterized in that: The request type is one of a page operation request, an application start request, and a semantic search request; If the request type is a page operation request, the formatted data further includes an index parameter, where the index parameter is an identifier that uniquely identifies the target element pointed to by the request data.

10. An electronic equipment control system, characterized in that: The system comprises: A device terminal, used for creating a direct connection group, wherein the device terminal is independent of the electronic device or integrated in the electronic device; A control terminal, configured to establish a communication connection with the device terminal according to the direct connection group; if the communication connection is successfully established, capture user audio data and convert the user audio data into text data; parse the text data into formatted data, the formatted data including a request type identifier and request data, the request type identifier being determined according to a request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; encapsulate the formatted data into a voice control instruction, the voice control instruction carrying the request type identifier and the request data; and send the voice control instruction to the device terminal; The device terminal is also used to receive the voice control instruction from the control terminal; determine the target request type of the voice control instruction according to the request type identifier in the voice control instruction and the first mapping relationship; determine the target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, the second mapping relationship being a one-to-one correspondence between the request type and the processing class; call the target processing class and operate the electronic device according to the request data in the voice control instruction.

Images