Electronic device control method and system

Abstract

The present application provides an electronic device control method and system. Among them, the method is applied to a device terminal and includes: receiving a voice control instruction from a control terminal, establishing a communication connection between the device terminal and the control terminal by creating a direct connection group, and the voice control instruction carries a request type identifier and request data; determining the target request type of the voice control instruction according to the request type identifier and a preset first mapping relationship, where the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type; determining the target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, where the second mapping relationship is a one-to-one correspondence between the request type and the processing class; calling the target processing class and operating the electronic device according to the request data. Through the microphone-free voice adaptation solution, voice control of the electronic device can be achieved without relying on a microphone, which can not only reduce the device cost but also not affect the user experience.

Description

Technical Field

[0001] This application relates to the technical field of device voice control, and particularly to an electronic device control method and system. Background Art

[0002] With the progress of technology and the continuous growth of user needs, the intelligence of electronic devices has become the current development trend. In this trend, voice interaction has also become an essential part of reflecting the intelligence of electronic devices. In intelligent devices, users can control various device functions through voice commands, such as navigation, entertainment systems, phone calls, etc., greatly improving the user experience.

[0003] In the related art, in order to implement the voice interaction function of an electronic device, it is necessary to configure a high-quality microphone device to ensure the accuracy and response speed of voice recognition. However, for electronic devices that do not have a microphone installed or have a malfunctioning or poorly performing microphone in order to reduce costs, this will directly affect the use experience of intelligent devices and reduce user satisfaction. Summary of the Invention

[0004] In view of the above-mentioned disadvantages of the prior art, this application discloses an electronic device control method and system for solving the technical problem of poor user experience in the intelligent operation of electronic devices in the prior art.

[0005] In a first aspect, this application provides an electronic device control method, which is applied to a device terminal that is independent of or integrated into the electronic device. The method includes: receiving a voice control instruction from a control terminal, establishing a communication connection with the control terminal by the device terminal creating a direct connection group, and the voice control instruction carries a request type identifier and request data; determining the target request type of the voice control instruction according to the request type identifier and a preset first mapping relationship, where the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type; determining the target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, where the second mapping relationship is a one-to-one correspondence between the request type and the processing class; calling the target processing class and operating the electronic device according to the request data.

[0006] In an embodiment of this application, the target request type is one of a page operation request, an application opening request, and a semantic search request; the target processing class is one of a page information processing class, an application opening processing class, and a semantic search processing class.

[0007] In one embodiment of the present application, the calling of the target processing class to operate the electronic device according to the request data includes: if the target request type is a page operation request, obtaining a page information processing class, where the page information processing class includes an accessibility service class and a page root element processing class, and a root element obtaining method is defined in the accessibility service class and a page root element listening interface is implemented; calling the page root element processing class to create a root element listening instance, and calling the root element obtaining method to obtain the page root element, and calling back and assigning the page root element to the root element listening instance; operating the electronic device according to the assigned root element listening instance and the request data.

[0008] In one embodiment of the present application, the operating of the electronic device according to the assigned root element listening instance and the request data includes: calling the element finding method in the page root element processing class, and traversing the page root element and each sub-element corresponding to the page root element in the element finding method; if there is a target element matching the request data, triggering a click event of the target element to operate the electronic device.

[0009] In one embodiment of the present application, the determining method of the target element includes: comparing an index parameter with a preset parameter threshold, where the index parameter is carried in the voice control instruction, and the index parameter is an identifier uniquely identifying the target element referred to by the request data; if the index parameter is greater than the parameter threshold, during the process of traversing the page root element and the sub-elements, recording the number of times of traversing the element that conforms to the request data, and when the number of times conforms to the index parameter, determining the element corresponding to the number of times as the target element.

[0010] In one embodiment of the present application, the calling of the target processing class to operate the electronic device according to the request data includes: if the target request type is a semantic search request, obtaining a semantic search processing class; calling the semantic search processing class to obtain a data parsing method and an activity starting method; using the data parsing method to parse the request data to obtain an activity instance, where the activity instance includes a target application package name and an activity name; passing the activity instance to the activity starting method to trigger an activity starting event to operate the electronic device.

[0011] In an embodiment of the present application, before receiving the voice control instruction from the control terminal, it further includes: detecting whether the electronic device has a voice capture and recognition function; if the electronic device does not have the voice capture and recognition function, sending a voice control instruction through the control terminal to control the electronic device; if the electronic device has the voice capture and recognition function, detecting whether there is a fault in the voice capture and recognition function, and in the case where there is a fault in the voice capture and recognition function, sending a voice control instruction through the control terminal to control the electronic device.

[0012] In a second aspect, the present application provides an electronic device control method, which is applied to a control terminal. The method includes: if the communication connection with the device terminal is successfully established, capturing user audio data and converting the user audio data into text data. The communication connection is established through a direct connection group created by the device terminal, and the device terminal is independent of the electronic device or integrated in the electronic device; parsing the text data into formatted data, where the formatted data includes a request type identifier and request data, and the request type identifier is determined according to the request type of the text data and a first mapping relationship, and the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type; encapsulating the formatted data into a voice control instruction, where the voice control instruction carries the request type identifier and the request data; sending the voice control instruction to the device terminal so that the device terminal operates the electronic device according to the voice control instruction.

[0013] In an embodiment of the present application, the request type is one of a page operation request, an application opening request, and a semantic search request; if the request type is a page operation request, the formatted data further includes an index parameter, and the index parameter is an identifier that uniquely identifies the target element referred to by the request data.

[0014] In a third aspect, the present application provides an electronic device control system, the system comprising: a device terminal for creating a direct connection group, the device terminal being independent of the electronic device or integrated in the electronic device; a control terminal for establishing a communication connection with the device terminal according to the direct connection group; if the communication connection is successfully established, capturing user audio data and converting the user audio data into text data; parsing the text data into formatted data, the formatted data including a request type identifier and request data, the request type identifier being determined according to the request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; encapsulating the formatted data into a voice control instruction, the voice control instruction carrying the request type identifier and the request data; sending the voice control instruction to the device terminal; the device terminal is further configured to receive the voice control instruction from the control terminal; determining the target request type of the voice control instruction according to the request type identifier in the voice control instruction and the first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; determining the target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, the second mapping relationship being a one-to-one correspondence between the request type and the processing class; calling the target processing class and operating the electronic device according to the request data in the voice control instruction.

[0015] As described above, an electronic device control method and system provided by an embodiment of the present application have the following beneficial effects:

[0016] The device terminal establishes a communication connection with the control terminal by creating a direct connection group to receive a voice control instruction from the control terminal. The voice control instruction carries a request type identifier and request data. Then, according to the request type identifier and a preset first mapping relationship, the target request type of the voice control instruction is determined. The first mapping relationship is a one-to-one correspondence between the request type identifier and the request type. Next, according to the target request type and a preset second mapping relationship, the target processing class of the voice control instruction is determined. The second mapping relationship is a one-to-one correspondence between the request type and the processing class. Finally, the target processing class is called to operate the electronic device according to the request data. Based on the control terminal capturing the user voice and controlling the electronic device by receiving the voice control instruction transmitted by the control terminal, and for different voice control requests, different processing classes are configured for the device terminal, which can meet the precise execution of different voice control requests. In this way, through the microphone-free voice adaptation scheme, the voice control of the electronic device is realized without relying on a microphone, which not only reduces the device cost but also improves the user experience.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and should not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application. Obviously, the drawings in the following description are only some embodiments of this application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. In the drawings:

[0019] Figure 1 is a schematic diagram of an implementation environment of an electronic device control system shown in an exemplary embodiment of this application;

[0020] Figure 2 is a schematic diagram of another implementation environment of an electronic device control system shown in an exemplary embodiment of this application;

[0021] Figure 3 is a flowchart of an electronic device control method shown in an exemplary embodiment of this application;

[0022] Figure 4 is a class diagram of page information processing shown in an exemplary embodiment of this application;

[0023] Figure 5 is a class diagram of application startup processing shown in an exemplary embodiment of this application;

[0024] Figure 6 is a class diagram of semantic search processing shown in an exemplary embodiment of this application;

[0025] Figure 7 is a class diagram of a device terminal shown in an exemplary embodiment of this application;

[0026] Figure 8 is a flowchart of another electronic device control method shown in an exemplary embodiment of this application;

[0027] Figure 9 is a class diagram of a control terminal shown in an exemplary embodiment of this application;

[0028] Figure 10 is a flowchart of a specific electronic device control method shown in an exemplary embodiment of this application;

[0029] Figure 11 is a block diagram of an electronic device control system shown in an exemplary embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The embodiments of the present application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application and not for limiting the protection scope of the present application.

[0031] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0032] In the following description, a large number of details are explored to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0033] In a smart device, in order to implement the voice interaction function, it needs to be configured with a high-quality microphone device to ensure the accuracy and response speed of voice recognition and improve the user experience. However, through the research of the inventors of the present application, it is found that the configuration and maintenance of the microphone pose a significant challenge to cost-effectiveness. On the one hand, the installation of high-quality microphones increases the hardware cost of the device and compresses the profit margin; on the other hand, if the microphone fails or its performance is poor, it will directly affect the use experience of the smart device and reduce user satisfaction.

[0034] Therefore, please refer to Figure 1 , Figure 1 which is a schematic diagram of the implementation environment of an electronic device control system shown in an exemplary embodiment of the present application. As Figure 1 shown, this implementation environment includes a control terminal 110, a device terminal 120, and an electronic device 130. Among them, the control terminal 110 is independent of the electronic device 130, the device terminal 120 is integrated in the electronic device, and the control terminal 110 and the device terminal 120 are used to implement the voice control of the electronic device 130. In addition, please refer to Figure 2 , Figure 2 which is a schematic diagram of another implementation environment of an electronic device control system shown in an exemplary embodiment of the present application. As Figure 2As shown, the implementation environment includes a control terminal 210, a device terminal 220, and an electronic device 230. Among them, the control terminal 210 is independent of the electronic device 230, and the device terminal 220 is also independent of the electronic device 130. The control terminal 110 and the device terminal 120 are used to implement voice control of the electronic device 130. In the electronic device control system, based on the control terminal capturing the user's voice, parsing and encapsulating the user's voice, and transmitting the voice control instruction to the device terminal, the voice control of the electronic device is realized. The control terminal can support diverse user voices. In the device terminal, different processing classes are configured for voice control instructions of different request types, which can meet the precise execution of different voice control requests. In this way, through the microphone-free voice adaptation scheme, the voice control of the electronic device is realized without relying on a microphone, which not only reduces costs but also improves the user experience.

[0035] Among them, the electronic device can be an electrical appliance, such as a television, a refrigerator, a washing machine, an air conditioner, and a floor cleaning robot, etc., and the usage scenarios of the electrical appliances are not restricted, such as household appliances, vehicle-mounted electrical appliances, etc. It can be an entertainment electronic device, such as an audio system, a player, and a game console, etc. It can be an industrial electronic device, or a car infotainment system. The electronic device can have a voice capture and recognition function or not have a voice capture and recognition function. The control terminal can be a device such as a mobile phone, a tablet, a laptop computer, a tablet computer, a smart watch, and a smart bracelet with a voice capture and recognition function.

[0036] Exemplarily, in Figure 1 's implementation environment, the control terminal 110 can be a mobile phone, the device terminal 120 can be a vehicle-mounted terminal, and the electronic device 130 can be a car infotainment system. That is, by capturing the user's voice with the mobile phone, parsing and encapsulating the user's voice, and transmitting the voice control instruction to the vehicle-mounted terminal, the voice control of the car infotainment system is realized; in Figure 2 's implementation environment, the control terminal 210 can be a mobile phone, the device terminal 220 can be a vehicle-mounted terminal, and the electronic device 230 can be a vehicle-mounted air conditioner. That is, by capturing the user's voice with the mobile phone, parsing and encapsulating the user's voice, and transmitting the voice control instruction to the vehicle-mounted terminal, the voice control of the vehicle-mounted air conditioner is realized.

[0037] Please refer to Figure 3 , Figure 3 which is a flowchart of an electronic device control method shown in an exemplary embodiment of the present application. This method can be applied to the implementation environment shown in Figure 1 or Figure 2 It should be understood that this method can also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment applicable to this method.

[0038] In addition, the electronic device control method can be applied to device terminals under various operating systems, including but not limited to device terminals under the Android operating system.

[0039] As Figure 3 shown, in an exemplary embodiment, the electronic device control method at least includes steps S310 to S340, which are introduced in detail as follows:

[0040] Step S310: Receive a voice control instruction from a control terminal. The device terminal establishes a communication connection with the control terminal by creating a direct connection group. The voice control instruction carries a request type identifier and request data.

[0041] Among them, the voice control instruction refers to an instruction in which the control terminal parses and encapsulates the captured user voice instruction into formatted data. The voice control instruction includes a request type identifier field and a request data field. The request type identifier field describes the request type identifier, and the request type identifier represents the request type of the voice control instruction; the request data field describes the request data, and the request data represents the request purpose of the voice control instruction, that is, the user intention.

[0042] Exemplarily, use "action":"..." as the request type identifier field and "data":"..." as the request data field.

[0043] In a possible embodiment, the voice control instruction further includes a communication identifier field, which is used to identify a communication between the device terminal and the control terminal. In this way, it is convenient to track the communication connection process between the two and the flow path of the interaction information, monitor the entire life cycle of the communication, and facilitate problem troubleshooting and log analysis.

[0044] Exemplarily, use "requestId":"..." as the communication identifier field.

[0045] In addition, when two devices are connected through P2P (Peer-to-Peer), two main roles are defined: P2P Group Owner (abbreviated as GO, that is, the group owner) and P2P Group Client (abbreviated as GC, that is, the group member). The device terminal creates a direct connection group. As the GO, it is equivalent to a server, and the device connected at the other end is the GC, which is equivalent to a client. In this embodiment, the device terminal is the GO, that is, the server, and the control terminal is the GC, that is, the client. Other devices can join the group by connecting to the GO device, but cannot directly connect to the GC device.

[0046] In one embodiment, the method for establishing a communication connection includes: obtaining and invoking a first direct connection management class to create a direct connection group, enabling the control terminal to search for and access the group owner of the direct connection group, where the group owner is a device terminal; creating a server management class, which includes member variables such as a server socket, a thread pool object, and a server startup method is defined, and a connection acceptance method is defined in the server startup method; in the thread pool, invoking the server startup method to create a socket instance and execute the connection acceptance method to listen for the access of the control terminal; if the access of the control terminal is detected, a communication connection is established.

[0047] Among them, the first direct connection management class is pre-configured on the device terminal.

[0048] Exemplarily, the first direct connection management class is the Wi-Fi direct connection management class WifiP2pManager. That is, WifiP2pManager under the Wifi p2p (Wi-Fi Peer-to-Peer) technology (such as in a device terminal under the Android operating system, it is the WifiP2pManager under the Android wifi p2p technology, that is, the Android Wi-Fi Peer-to-Peer technology). The direct connection group is the wifi p2p Group (Wi-Fi Peer-to-Peer Group).

[0049] Exemplarily, the method for creating a direct connection group of the device terminal is to obtain the first direct connection management class WifiP2pManager according to the direct connection management class acquisition method provided by the operating system, and after initializing by invoking the initialization method initialize() of the first direct connection management class WifiP2pManager, create a channel Channel to obtain a channel instance, that is, a Channel instance, and then invoke the direct connection group creation method createGroup() of the first direct connection management class WifiP2pManager and pass in the Channel instance to create a direct connection group and become the group owner.

[0050] In this embodiment, the ServerManager class is used to manage the connection of device terminals. This class includes member variable ① the server socket ServerSocket, which is used to transfer data with the control terminal, and member variable ② the thread pool object ExecutorService, which is used to provide sub-threads to send and receive data with the control terminal. This class defines method ① the server startup method startServer(), which is used to start the device terminal to wait for the connection of the control terminal. Among them, the connection acceptance method serverSocket.accept() is defined in the server startup method startServer(), which is used to accept the access of the device terminal and establish a communication connection.

[0051] In this embodiment, after the direct connection group is successfully created, a server management instance, that is, a ServerManager instance, will be created first to call the server startup method startServer() to create a socket instance, that is, a ServerSocket instance. In this ServerSocket instance, the port number is specified, and then the server startup method startServer() calls the connection acceptance method serverSocket.accept() to wait for the access of the control terminal. If the access of the control terminal is monitored, a communication connection will be established.

[0052] In this embodiment, through the direct connection communication technology and the way of creating the ServerManager class, the communication connection between the device terminal and the control terminal is realized. In this way, a connection can be quickly established between the device terminal and the control terminal, and there is no need to pre-configure the network environment, improving the user experience.

[0053] In one embodiment, the receiving method of the voice control instruction includes: calling the client processing class to create a client processing instance. The client processing class is included in the inner class of the ServerManager class, and a first listening method is defined in the client processing class; through the client processing instance, the socket instance is called back; the input stream acquisition method is called by using the socket instance to obtain the input stream; the first listening method is called, and the voice control instruction is obtained through the input stream.

[0054] In this embodiment, the ServerManager class also includes an inner class, that is, the ClientHandler class, which is used to handle the interaction with the control terminal. The method ① the first listening method startListening() is defined in the ClientHandler class, which is used to listen to the data transmitted from the device terminal.

[0055] In addition, the connection acceptance method serverSocket.accept() is a blocking method, and this method will return a ServerSocket instance when the control terminal accesses.

[0056] In this embodiment, after returning the ServerSocket instance, the client processing class ClientHandler is called to create a client processing instance, that is, the ClientHandler instance, as a member variable ③ ClientHandler instance of the server management class ServerManager, and through the ClientHandler instance, it is called back to the ServerSocket instance. The input stream acquisition method socket.getInputStream() (provided by the operating system application programming interface) is used by the ServerSocket instance to obtain the input stream inputStream. This input stream inputStream can obtain the data transmitted by the device terminal. At the same time, it is used as a member variable ① inputStream of the client processing class ClientHandler. Finally, the first listening method startListening() is called to obtain the voice control instruction through the input stream inputStream.

[0057] In this embodiment, the client processing class is called to create a client processing instance, and based on the client processing instance, the socket instance is called back and the first listening method is called to listen for and obtain the voice control instruction through the inner class and the callback mechanism. In this way, it can ensure that the voice control instruction is received and processed in a timely manner.

[0058] In a possible embodiment, the client processing class ClientHandler further includes a member variable ② boolean variable isListening, that is, the boolean variable listening state isListening, which is used to mark whether to continue listening for the data sent by the control terminal.

[0059] As a possible embodiment, the startListening() method of the client processing class ClientHandler determines whether to continue listening for data sent by the control terminal based on the listening state isListening. If it is determined according to the listening state isListening that it is necessary to continue listening for data sent by the control terminal, the input stream reading method inputStream.readLine() (provided natively by the operating system) is called to continuously read the data in the input stream inputStream. The inputStream.readLine() method is a blocking method and will only continue to execute when data is sent by the control terminal. Its return value is the data sent by the control terminal, and the data read is of string type. If it is determined according to the listening state isListening that it is not necessary to continue listening for data sent by the control terminal, the client processing class ClientHandler does not perform any processing.

[0060] Step S320: Determine the target request type of the voice control instruction according to the request type identifier and the preset first mapping relationship. The first mapping relationship is a one-to-one correspondence between the request type identifier and the request type.

[0061] Among them, the first mapping relationship is pre-configured, and the relationship between the request type identifier and the request type can be a one-to-one correspondence.

[0062] In addition, the request type refers to the type of user intention represented by the voice control instruction.

[0063] Step S330: Determine the target processing class of the voice control instruction according to the target request type and the preset second mapping relationship. The second mapping relationship is a one-to-one correspondence between the request type and the processing class.

[0064] Among them, the second preset relationship is pre-configured, and the relationship between the request type and the processing class can be a one-to-one correspondence.

[0065] In addition, the processing class refers to the implementation of the specific operation logic of the voice control instruction.

[0066] In one embodiment, the target request type is one of a page operation request, an application opening request, and a semantic search request; the target processing class is one of a page information processing class, an application opening processing class, and a semantic search processing class.

[0067] Exemplarily, if the target request type is a page operation request, the target processing class is a page information processing class; if the target request type is an application opening request, the target processing class is an application opening processing class; if the target request type is a semantic search request, the target processing class is a semantic search processing class.

[0068] Exemplarily, if the request type identifier is a page operation request identifier, the target request type is a page operation request, and the corresponding target processing class is the page information processing class NodeInfoHolder; if the request type identifier is an application open request identifier, the target request type is an application open request, and the corresponding target processing class is the application open processing class OpenAppManager; if the request type identifier is a semantic search request identifier, the target request type is a semantic search request, and the corresponding target processing class is the semantic search processing class SchemaHanlder.

[0069] In this embodiment, to improve the convenience of voice control of intelligent electronic devices, that is, users can input various semantic information for controlling device operations. Therefore, the user intentions corresponding to the user's voice information are classified, that is, divided according to the logic of the device terminal executing voice control instructions, and the request types corresponding to the voice control instructions are divided into page operation requests, application opening requests, and semantic search requests. Among them, the page operation request is an operation request for the page root element control, the application opening request is an operation request for opening an application, and the semantic search request is an operation request for obtaining and executing device activities based on semantic parsing other than the operation request for the page root element control and the operation request for opening an application.

[0070] Exemplarily, the page operation request is applicable to the scenario where what you see on the device interface can be spoken. For example, words such as "start playing" and "ok" that the user sees on the device interface. The user can input voice based on the page root element to control the terminal to transmit a voice control instruction of the page operation request type; the application opening request is applicable to the opening of an application. For example, the user inputs the voice "open a certain music software", and the control terminal transmits a voice control instruction of the application opening request type; the semantic search request is applicable to device operations other than the above two requests, that is, the voice input by the user is something the user says casually, which is neither the page root element nor opening a certain application. For example, the user inputs the voice "I want to listen to songs by a certain (singer's name)", and the control terminal transmits a voice control instruction of the semantic search request type.

[0071] Exemplarily, for the voice control instructions of the above three request types, the "action" for the page operation request and the application opening request is fixed. For example, their "action" is "word" and "openApp" respectively, while the "action" in the words spoken by the user is diverse. For example: "music" (I want to listen to the songs of xx), "sms" (send a text message to xx), etc. Therefore, the "action" of the semantic search request is defined as "semantic". After receiving the request type identifier of the voice control instruction from the control terminal, it will first determine whether its "action" is "word" or "openApp", and then confirm whether it is "semantic".

[0072] In this embodiment, different processing classes are configured according to the voice control instructions of different request types. On the one hand, it not only meets various voice control scenarios of various intelligent electronic devices and improves the convenience of voice control of intelligent electronic devices. On the other hand, each processing class only needs to focus on the request type it is responsible for. In this way, the response speed of the electronic device to the voice control instruction is improved, ensuring that the user can get accurate and timely feedback, thereby enhancing the overall user experience. In addition, the maintainability of the processing class is enhanced, making it easier to locate the error source and perform targeted repairs when problems occur. And when it is necessary to add voice control instructions of a new request type or modify the processing logic of existing instructions, only the corresponding processing class needs to be updated or added, and its scalability is enhanced.

[0073] Step S340, call the target processing class and operate the electronic device according to the request data.

[0074] In this embodiment, after determining the target processing class corresponding to the voice control instruction, the request data will be distributed to the target processing class, and the target processing class operates the electronic device according to the request data to complete the response to the voice control instruction.

[0075] In one embodiment, calling the target processing class and operating the electronic device according to the request data includes: if the target request type is a page operation request, obtain a page information processing class. The page information processing class includes an accessibility service class and a page root element processing class. The accessibility service class defines a root element obtaining method and implements a page root element listening interface; call the page root element processing class to create a root element listening instance, and call the root element obtaining method to obtain the page root element, and call back and assign the page root element to the root element listening instance; operate the electronic device according to the assigned root element listening instance and the request data.

[0076] In this embodiment, if the target request type is a page operation request (i.e., action = word), the execution logic of the request data in the voice control instruction is as follows: The request data is passed to the page information processing class NodeInfoHolder to operate the electronic device, that is, to perform the page operation of the electronic device.

[0077] In this embodiment, the page information processing class NodeInfoHolder includes the accessibility service class MyAccessibilityService and the page root element processing class NodeInfoHandler. The accessibility service class MyAccessibilityService defines a root element acquisition method getRootInActiveWindow() and implements the page root element listening interface INodeInfoListener.

[0078] Exemplarily, the page root element listening interface INodeInfoListener defines a page root element acquisition method getNodeInfo(). This interface builds a bridge between the accessibility service class MyAccessibilityService and the page root element processing class NodeInfoHandler.

[0079] In this embodiment, the accessibility service class MyAccessibilityService inherits from AccessibilityService (the accessibility service provided by the operating system). After guiding the user to enable the accessibility service function through the standard method provided by the operating system, the page root element processing class NodeInfoHandler creates a root element listening instance. The accessibility service class MyAccessibilityService obtains the current page root element through its internal getRootInActiveWindow() method. At the same time, the accessibility service class MyAccessibilityService implements the page root element listening interface INodeInfoListener. This interface is responsible for calling back the page root element obtained by the accessibility service class MyAccessibilityService to the page root element processing class NodeInfoHandler to implement the assignment of the root element listening instance. At this time, after the root element listening instance is assigned, the page root element processing class NodeInfoHandler can obtain the page root element. At the same time, this class can also obtain the request data, and thus can implement the page operation logic according to the page root element and the elements in the request data.

[0080] In this way, by introducing an accessibility service class, a page root element processing class, and a page root element listening interface, accessibility service function support is provided for page operations. Through the effective management of the page root element and the listening of request data, an efficient response to voice control instructions for page operation requests is achieved, enhancing the overall user experience.

[0081] In one embodiment, operating an electronic device based on the assigned root element listening instance and request data includes: calling the findElement method in the page root element processing class, and traversing the page root element and its corresponding child elements in the findElement method; if a target element matching the request data exists, triggering a click event of the target element to operate the electronic device.

[0082] Exemplarily, the findElement method includes a public method findTextInNode(string: String, index: Int) and a private method findTextInNode(nodeInfo: AccessibilityNodeInfo).

[0083] Exemplarily, traversing the page root element and its corresponding child elements in the findElement method includes: calling the public method to obtain the request data and the page root element in the root element listening instance, and passing them to the private method; calling the private method to traverse the page root element and its corresponding child elements.

[0084] In this embodiment, the page root element processing class NodeInfoHandler defines a public method findTextInNode(string: String, index: Int) and a private method findTextInNode(nodeInfo: AccessibilityNodeInfo). Among them, the public method findTextInNode(string: String) is called when the request data is distributed to obtain the request data. Its parameter string is the keyword spoken by the user, such as "OK". Then, within this public method, the page root element is obtained through the iNodeInfoListener instance and passed to the private method findTextInNode(nodeInfo: AccessibilityNodeInfo). In this private method, the root element and its child elements are recursively traversed. If a target element matching the request data is found, a click event of the target element is triggered to perform the page operation.

[0085] Exemplarily, triggering a click event of the target element means executing the click method provided by the operating system to achieve the click of the target element.

[0086] In this embodiment, a method for finding an element is defined in the page root element processing class. By traversing the root element and its child elements in combination with the request data, accurate positioning of the target element corresponding to the request data is achieved, ensuring the reliability of page operations.

[0087] In one embodiment, the method for determining a target element includes: comparing an index parameter with a preset parameter threshold. The index parameter is carried in the voice control instruction and is an identifier that uniquely identifies the target element referred to by the request data. If the index parameter is greater than the parameter threshold, during the process of traversing the page root element and its child elements, record the number of times an element that matches the request data is traversed, and when the number of times matches the index parameter, determine the element corresponding to the number of times as the target element.

[0088] Exemplarily, if the index parameter is less than or equal to the parameter threshold, then determine the first element that matches the request data traversed as the target element.

[0089] In this embodiment, considering that there are multiple elements with the same keyword in the current page root element and its child elements. For example, there are multiple "OK"s. During the process of encapsulating the voice control instruction in the control terminal, if the request type is a page operation request, then when encapsulating the voice control instruction, an index parameter field is set, and the index parameter field describes the index parameter. Therefore, the voice control instruction received by the electronic device also carries the index parameter. Among them, this index parameter is used to identify the target element among multiple elements when there are multiple elements with the same keyword and the keyword of the element represented by the request data happens to be this word.

[0090] Exemplarily, use "index":"..." as the request type identification field, where "..." is a numerical value.

[0091] Exemplarily, when there are three elements with the keyword "OK", these three "OK"s correspond to different indexes. The index corresponding to the first "OK" is 0, the index corresponding to the second "OK" is 1, and the index corresponding to the third "OK" is 2. If the user wants to click on the second "OK", then index = 1.

[0092] In this embodiment, before traversing the root element and its child elements, compare the index parameter with the preset parameter threshold. If the index parameter is less than or equal to the parameter threshold, then during the traversal, determine the first element that matches the request data traversed as the target element. If the index parameter is greater than the parameter threshold, then during the traversal, record the number of times an element that matches the request data is traversed, and when the number of times matches the index parameter, determine the element corresponding to the number of times as the target element.

[0093] Exemplarily, the parameter threshold is set to 0. When the value of index is less than or equal to 0 (less than is for fault tolerance), when traversing to the first element that matches the requested data, the click event is triggered; when index is greater than 0, the number of occurrences of the element that matches the requested data is recorded, and when the number of occurrences matches the value of index, the click event is triggered; if after traversing all root elements and their child elements, the recorded number still does not match the value of index, it is regarded as a failure.

[0094] Exemplarily, if the keyword in the requested data is "OK", when index = 0, when traversing to the first "OK", the click event is triggered, and when index = 3, when traversing to the fourth "OK", the click event is triggered.

[0095] As a possible embodiment, the judgment rule for the number of times to match the index parameter can be set according to requirements. For example, the judgment rule for the above "match" is: number of times = index parameter + 1. Of course, the judgment rule for "match" can also be: number of times = index parameter + 2, number of times = index parameter, or number of times = index parameter - 1, etc. The embodiments of the present application do not make specific limitations.

[0096] In this embodiment, by introducing the index parameter and the threshold mechanism, when there are multiple elements on the page with the same keyword, the target element can be determined more accurately, avoiding the possibility of misoperation, and further ensuring the reliability of page operations.

[0097] Please refer to Figure 4 , Figure 4 which is a class diagram of page information processing shown in an exemplary embodiment of the present application. As Figure 4 shown, the page information processing class includes an accessibility service class and a page root element processing class. Among them, the accessibility service class inherits from the accessibility service provided by the operating system. The accessibility service defines a method for obtaining the current page root element. Therefore, the accessibility service class inherits this method. In addition, the implementation of obtaining the page root element in the accessibility service class is achieved by defining a page root element listening interface. When calling the page root element processing class, a root element listening instance can be created, and the page root element is passed through the page root element listening interface to assign the root element listening instance, and then the keyword in the requested data is obtained for element traversal.

[0098] In a possible embodiment, after triggering the click event of the target element and performing the page operation, it further includes: obtaining the page operation result; if the operation is successful, assembling the information of successful operation and returning it to the control terminal; if the operation fails, assembling the information of failed operation and returning it to the control terminal.

[0099] Exemplarily, the reasons for operation failure include: failure due to not traversing an element that matches the request data, and failure due to the click event not being achievable.

[0100] Exemplarily, when the operation is successful, the returned message is: {"responseId":"message":"Operation successful"}; when the operation fails: {"responseId":"message":"Operation failed":"Reason for failure"}.

[0101] In one embodiment, calling a target processing class to perform an operation on an electronic device according to request data includes: if the target request type is an application opening request, obtaining an application opening processing class; calling the application opening processing class to obtain a list of application information, and traversing the application names in the list of application information, where the list of application information includes application names and application package names; if there is a target application name that matches the request data, determining the target application package name from the list of application information according to the target application name, and triggering an application start event of the target application corresponding to the target application package name to perform an operation on the electronic device.

[0102] In this embodiment, if the target request type is an application opening request (i.e., action=openApp), the execution logic of the request data in the voice control instruction is: passing the request data to the application opening processing class OpenAppManager to perform an operation on the electronic device, that is, performing an application opening operation.

[0103] In this embodiment, the application opening processing class OpenAppManager obtains a list of application information installed on the electronic device, where the list of application information includes application names and application package names (the application names are used to match the request data, and the application package names are used to match the target application), and then traverses the application names in the list of application information. If there is a target application name that matches the request data, the target application package name is determined, and an application start event of the target application corresponding to the target application package name is triggered to perform an application opening operation.

[0104] Exemplarily, the application opening processing class OpenAppManager can obtain all the application information installed on the electronic device through the application information acquisition method PackageManager() provided by the operating system to generate a list of application information.

[0105] Exemplarily, triggering an application start event of the target application corresponding to the target application package name means performing the start method startActivity() provided by the operating system to start the target application.

[0106] In this embodiment, an application startup processing class is introduced to achieve an exact match of the application package name of the target application, so as to start the correct application, avoid misoperations in application startup, and ensure the reliability of the application opening operation.

[0107] Please refer to Figure 5 , Figure 5 which is a diagram of an application startup processing class shown in an exemplary embodiment of the present application. As Figure 5 shown, the application startup processing class realizes the opening operation of the target application through the application package name.

[0108] In a possible embodiment, after triggering the application startup event of the target application corresponding to the target application package name and executing the application opening operation, it further includes: obtaining the result of the application opening operation; if the operation is successful, assembling the information of successful operation and returning it to the control terminal; if the operation fails, assembling the information of failed operation and returning it to the control terminal.

[0109] Exemplarily, the reasons for the operation failure include: the failure of not traversing the application name that matches the request data, and the failure of the application startup event not being realized.

[0110] Exemplarily, when the operation is successful, the returned message is: {"responseId": "message": "Operation successful"}; when the operation fails: {"responseId": "message": "Operation failed": "Reason for failure"}.

[0111] In an embodiment, calling the target processing class to operate the electronic device according to the request data includes: if the target request type is a semantic search request, obtaining the semantic search processing class; calling the semantic search processing class to obtain the data parsing method and the activity startup method; using the data parsing method to parse the request data to obtain an activity instance, where the activity instance includes the target application package name and the activity name; passing the activity instance to the activity startup method to trigger the activity startup event and operate the electronic device.

[0112] In this embodiment, if the target request type is an application opening request (i.e., action = semantic), the execution logic of the request data in the voice control instruction is: passing the request data to the semantic search processing class SchemaHanlder to operate the electronic device, that is, executing the activity startup operation.

[0113] In this embodiment, the data parsing method Uri.parse(schema) and the activity start method startActivity() provided by the operating system are obtained through the semantic search processing class SchemaHanlder. First, the Uri.parse(schema) method is used to parse the request data to obtain an activity instance, which includes the target application package name and the activity name. Then, the activity instance is passed to startActivity() to trigger the activity start event and perform the activity start operation.

[0114] Exemplarily, the Uri.parse(schema) method is used to parse the request data to obtain a uniform resource identifier instance, that is, a Uri instance. Then, the Uri instance is passed to an intent instance, that is, an Intent instance, which is the final activity instance.

[0115] Exemplarily, the activity instance is passed to the activity start method to trigger the activity start event and perform the activity start operation, that is, the activity start method performs the activity start operation according to the target application package name and the activity name in the activity instance.

[0116] In this embodiment, by introducing the semantic search processing class, support is provided for more complex voice input, realizing voice control of intelligent electronic devices under complex voice input and improving the user experience.

[0117] Please refer to Figure 6 , Figure 6 which is a semantic search processing class diagram shown in an exemplary embodiment of the present application. As Figure 6 shown, the semantic search processing class realizes activity processing through request data.

[0118] In a possible embodiment, after passing the activity instance to the activity start method to trigger the activity start event and perform the activity start operation, it further includes: obtaining the activity start operation result; if the operation is successful, assembling the information of successful operation and returning it to the control terminal; if the operation fails, assembling the information of failed operation and returning it to the control terminal.

[0119] Exemplarily, the reasons for operation failure include: failure due to not traversing the application package name or activity name that matches the request data, and failure due to the failure to implement the activity start event.

[0120] Exemplarily, when the operation is successful, the returned message is: {"responseId":"message":"Operation successful"}; when the operation fails: {"responseId":"message":"Operation failed":"Reason for failure"}.

[0121] In one embodiment, after operating the electronic device according to the request data, it further includes: calling an output stream acquisition method using a socket instance to obtain an output stream; calling a server data sending method to execute a first data transmission method defined in the client processing class, writing the device operation result into the output stream and sending it to the control terminal. A server data sending method is also defined in the server management class.

[0122] In this embodiment, the client processing class ClientHandler also defines a method ② the first data transmission method sendMessage(message: String) for sending data to the control terminal; the server management class ServerManager also defines a method ② the server data sending method ServerManager.sendMessage(message: String) for calling the sendMessage(message: String) method of the client processing class ClientHandler to directly send data to the device terminal.

[0123] In this embodiment, an output stream is obtained by calling the output stream acquisition method socket.getOutputStream() (provided for the operating system application programming interface) using a ServerSocket instance. The input stream outputStream can obtain the data to be transmitted to the device terminal. At the same time, as a member variable ② of the client processing class ClientHandler, the output stream outputStream, and then the server data sending method ServerManager.sendMessage(message: String) is called to execute the first data transmission method sendMessage(message: String) defined in the client processing class ClientHandler, writing the device operation result into the output stream outputStream and sending it to the control terminal to realize the feedback of the device operation result.

[0124] Exemplarily, execute the first data transfer method defined in the client processing class, write the device operation result to the output stream and send it to the control terminal, including: using the first data transfer method sendMessage(message: String), call the data writing method outputStream.println(message) and the data sending method outputStream.flush() provided by the operating system (provided by the operating system); execute the outputStream.println(message) method to write the device operation result to the output stream outputStream; execute the outputStream.flush() method to send the output stream outputStream writing the device operation result to the control terminal.

[0125] In this embodiment, the instant feedback of the device operation result is implemented through the client processing class, improving the user's interaction experience.

[0126] In a possible embodiment, the server management class ServerManager also defines a method ③ connection closing method closeConnection(), and the client processing class ClientHandler also defines a method ③ stop listening method stopListening(), which is used to send data to the control terminal.

[0127] As a possible embodiment, the closeConnection() method of the server management class ServerManager calls the stopListening() method of the client processing class ClientHandler to determine whether to continue listening for the data sent by the control terminal. The implementation method is to determine whether to continue listening for the data sent by the control terminal according to the listening state isListening. If the listening state isListening is set to false, stop listening and call the closeConnection() method to close the connection with the control terminal. The implementation method is to call the method socket closing method socket.close() inside the closeConnection() method to close the connection. After the connection is closed, close the thread pool executorService.

[0128] Please refer to Figure 7 , Figure 7 is a device terminal class diagram shown in an exemplary embodiment of the present application. As Figure 7As shown, at the device terminal, the direct connection management class includes an initialization method for instantiating the direct connection management class, a direct connection group creation method for creating a direct connection group and passing in a channel instance, and a group removal method for removing the direct connection group; the channel class includes a close method for closing the channel after closing the connection; the server management class includes member variables server socket, thread pool object, and client processing instance, and includes a server start method, a server data sending method, and a connection closing method; the client processing class includes member variables input stream, output stream, and listening status, and includes a listening method, a data transmission method, and a stop listening method.

[0129] In one embodiment, before receiving a voice control instruction from a control terminal, it further includes: detecting whether the electronic device has a voice capture and recognition function; if the electronic device does not have a voice capture and recognition function, then controlling the electronic device by sending a voice control instruction through the control terminal; if the electronic device has a voice capture and recognition function, then detecting whether there is a failure in the voice capture and recognition function, and in the case of a failure in the voice capture and recognition function, controlling the electronic device by sending a voice control instruction through the control terminal.

[0130] In this embodiment, considering the problem of affecting the user experience in the case where the electronic device does not have a voice capture and recognition function or there is a failure in the voice capture and recognition function, it is pre-detected whether the electronic device has a voice capture and recognition function or whether there is a failure in the voice capture and recognition function, and in the case of not having a voice capture and recognition function or a failure in the voice capture and recognition function, the scheme for implementing device voice control based on the control terminal is adaptively started. In this way, the redundancy of the electronic device voice control is increased through the adaptive switching mechanism, and the user experience is improved.

[0131] Exemplarily, detecting whether there is a failure in the voice capture and recognition function includes: obtaining the running status of the microphone and the voice assistant, where the running status includes normal running and abnormal running; if the running status of the microphone and / or the running status of the voice assistant is abnormal running, then generating and popping up a prompt message, where the prompt message is whether to enable the device control scheme based on the control terminal, and enabling the electronic device control scheme under the condition of receiving an enabling instruction; if the running status of the microphone and the running status of the voice assistant are normal running, then determining the sound reception status of the microphone according to the voice recognition accuracy information and / or voice recognition efficiency information of the voice assistant, where the sound reception status includes normal sound reception and abnormal sound reception, and generating and popping up a prompt message under the condition that the sound reception status of the microphone is abnormal sound reception, and starting the electronic device control scheme under the condition of receiving an enabling instruction.

[0132] Among them, the operating status of the microphone and the voice assistant indicates whether they are available. That is, normal operation indicates availability, and abnormal operation indicates unavailability; the sound collection status of the microphone indicates whether the microphone collects sound accurately. That is, normal sound collection indicates accuracy, and abnormal sound collection indicates inaccuracy and impaired microphone performance.

[0133] In this embodiment, when the electronic device is configured with a microphone, considering the problems that affect the user experience such as microphone failure or poor performance, the operating status and the sound collection status of the microphone are judged in advance. And when the operating status of the microphone is abnormal or the sound collection status is abnormal, a solution for implementing device voice control based on the control terminal is adaptively started. In this way, the redundancy of the voice control of the electronic device is increased through the adaptive switching mechanism. When the microphone fails or has poor performance, the control terminal is used to implement the voice control of the electronic device, ensuring the stability of the device voice control function and improving the user experience.

[0134] Exemplarily, the operating status of the microphone is judged by the hardware operating status monitoring system. The specific judgment method can be implemented based on existing methods, and the operating status of the voice assistant can also be implemented based on existing methods, which will not be elaborated in the embodiments of this application.

[0135] Exemplarily, the sound collection status is determined according to the speech recognition accuracy information and / or the speech recognition efficiency information of the voice assistant. The speech recognition accuracy information includes the speech recognition accuracy, and the speech recognition efficiency information includes the speech recognition efficiency value. Among them, when determining according to the speech recognition accuracy information, if the speech recognition accuracy value is greater than or equal to the preset accuracy value, it is determined that the sound collection is normal, otherwise, it is abnormal. When determining according to the speech recognition efficiency information, if the speech recognition efficiency value is greater than or equal to the preset efficiency value, it is determined that the sound collection is normal, otherwise, it is abnormal. When determining according to the speech recognition accuracy information and the speech recognition efficiency information, the speech recognition accuracy value, the speech recognition efficiency value and their respective preset weights are calculated. If the calculation result is greater than or equal to the preset value, it is determined that the sound collection is normal, otherwise, it is abnormal.

[0136] Exemplarily, the sound collection status is determined based on the speech recognition accuracy value and / or the speech recognition efficiency value of the voice assistant for historical voice commands. Among them, the historical voice command refers to the voice control command directly captured by the microphone configured in the device during historical control. The speech recognition accuracy value is the average value of the recognition accuracies in the recent multiple recognitions in the historical recognition of the voice assistant, and the speech recognition efficiency value is the average value of the recognition efficiencies in the recent multiple recognitions in the historical recognition of the voice assistant. During the historical recognition process, the accuracy and efficiency of each recognition will be recorded. In addition, the specific confirmation method for the accuracy and efficiency of each recognition can be implemented based on existing methods, which will not be elaborated in the embodiments of this application.

[0137] In the above electronic device control method, the device terminal establishes a communication connection with the control terminal by creating a direct connection group to receive voice control instructions from the control terminal. The voice control instructions carry a request type identifier and request data. Then, according to the request type identifier and a preset first mapping relationship, the target request type of the voice control instruction is determined. The first mapping relationship is a one-to-one correspondence between the request type identifier and the request type. Next, according to the target request type and a preset second mapping relationship, the target processing class of the voice control instruction is determined. The second mapping relationship is a one-to-one correspondence between the request type and the processing class. Finally, the target processing class is called to operate the electronic device according to the request data. Based on the control terminal capturing the user's voice and receiving the voice control instructions transmitted by the control terminal, the electronic device is controlled. Moreover, for different voice control requests, the device terminal configures different processing classes, which can meet the precise execution of different voice control requests. In this way, through the microphone-free voice adaptation scheme, the voice control of the electronic device is realized without relying on a microphone, which not only reduces the cost but also improves the user experience.

[0138] Moreover, if the electronic device does not need to be configured with a microphone, it not only reduces the cost but also enables the voice control of the electronic device. For an electronic device configured with a microphone, a backup solution for voice control is provided, expanding the control range of the electronic device and making it easier for users who are temporarily unable to interact with the electronic device closely to use and operate the electronic device.

[0139] Please refer to Figure 8 , Figure 8 which is a flowchart of another electronic device control method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 or Figure 2 the implementation environment shown. It should be understood that this method can also be applicable to other exemplary implementation environments, and the present embodiment does not limit the implementation environment to which this method applies.

[0140] In addition, this electronic device control method can be applied to control terminals under various operating systems, including but not limited to control terminals under the Android operating system.

[0141] As Figure 8 shown, in an exemplary embodiment, the electronic device control method at least includes steps S810 to S840, which are introduced in detail as follows:

[0142] Step S810, if the communication connection with the device terminal is successfully established, capture the user audio data and convert the user audio data into text data. The communication connection is established through the direct connection group created by the device terminal, and the device terminal is independent of the electronic device or integrated into the electronic device.

[0143] Exemplarily, by integrating various third-party toolkits such as the speech recognition services of iFlytek, Google, and Baidu, the user audio data is captured according to the third-party toolkit and converted into text data.

[0144] In one embodiment, the method for establishing a communication connection includes: obtaining a second direct connection management class and creating a broadcast receiver, where the second direct connection management class defines a group owner search method and a connection method; calling the group owner search method to search for group owners within a preset distance range, and using the broadcast receiver to listen for and callback group owner information; if a connection event is triggered, creating a direct connection instance according to the group owner information of the target group owner; calling the connection method to access the direct connection group according to the direct connection instance and establish a communication connection.

[0145] Among them, the second direct connection management class is pre-configured on the control terminal.

[0146] Exemplarily, the first direct connection management class is the Wi-Fi direct connection management class WifiP2pManager.

[0147] In this embodiment, the second direct connection management class WifiP2pManager is obtained according to the direct connection management class acquisition method provided by the operating system, and after initializing by calling the initialization method initialize() of the second direct connection management class WifiP2pManager, a channel Channel is created to obtain a channel instance, that is, the Channel instance, and a broadcast receiver ClientBroadcastReceiver is created to listen for and callback group owner information. Among them, the second direct connection management class WifiP2pManager defines a group owner search method wifiP2pManager.discoverPeers() and a connection method wifiP2pManager.connect().

[0148] In this embodiment, after setting up the listening, call wifiP2pManager.discoverPeers() to implement the search for group owners (device terminals) (the device terminal can only be searched by the control terminal after successfully creating a direct connection group). During the search, the broadcast receiver ClientBroadcastReceiver is used to listen for and callback group owner information (such as Internet protocol address, name, etc.) to obtain a group owner list. When searching for and connecting to a device terminal, the Channel instance is passed in, and a direct connection instance, that is, the WifiP2pConfig instance, is created according to the Internet protocol address of the device terminal. Finally, the WifiP2pConfig instance is passed to the wifiP2pManager.connect() method to connect to the direct connection group and establish a communication connection.

[0149] Exemplarily, the broadcast receiver ClientBroadcastReceiver can also monitor changes in the group owner and the connection status with each group owner.

[0150] Exemplarily, when the connection is successful, the monitoring by the broadcast receiver ClientBroadcastReceiver will call back the device terminal host information InetAddress.

[0151] In this embodiment, through the direct connection communication technology and the way of creating a broadcast receiver, the communication connection between the control terminal and the device terminal is realized. In this way, a connection can be quickly established between the control terminal and the device terminal without pre-configuring the network environment, improving the user experience.

[0152] In one embodiment, after the communication connection is successfully established, it further includes: calling back the host information of the device terminal; creating a socket management class, in which a second monitoring method and a second data transmission method are defined. The second monitoring method is used to monitor the device operation results from the device terminal according to the host information, and the second data transmission method is used to send voice control instructions to the device terminal according to the host information.

[0153] In this embodiment, after the communication connection is successfully established, it is necessary to implement the interaction with the device terminal, that is, sending data (mainly voice control instructions) and receiving data (mainly device operation results), which is realized by creating a socket management class SocketManager, responsible for managing the socket of the control terminal, and the host information InetAddress of the device terminal to be called back.

[0154] Among them, the socket management class SocketManager defines method ① the second monitoring method listenForMessages() and method ② the second data transmission method sendMessage(message:String). By calling the listenForMessages() method, the device operation results from the device terminal are monitored according to the host information. By calling the sendMessage(message:String) method, voice control instructions are sent to the device terminal according to the host information.

[0155] Exemplarily, through the sendMessage(message:String) method, the data writing method outputStream.println(message) and the data sending method outputStream.flush() (provided by the operating system) are called to implement the sending of voice control instructions.

[0156] In this embodiment, by creating a socket management class and implementing the interaction with the device terminal based on the second listening method and the second data transmission method of the socket management class, that is, sending voice control instructions and receiving device operation results, in this way, the instant sending and receiving of voice control instructions and device operation results can be ensured.

[0157] In a possible embodiment, the socket management class SocketManager includes member variable ① the socket Socket, which is responsible for transmitting data with the device terminal and is initialized through the host information InetAddress of the device terminal and the port number consistent with the device terminal; member variable ② the input stream inputStream, which is obtained through the input stream acquisition method socket.getInputStream() provided natively by the operating system, and this input stream inputStream can obtain the data transmitted by the device terminal; member variable ③ the output stream outputStream, which is obtained through the output stream acquisition method socket.getOutputStream() provided natively by the operating system, and this output stream outputStream can transmit data to the device terminal.

[0158] Exemplarily, the listenForMessages() method reads the data transmitted by the device terminal according to whether the socket Socket is connected. If it is still connected, it calls the input stream acquisition method inputStream.readLine() (provided natively by the operating system) to continuously read the data in the input stream inputStream. If the socket has been disconnected, the control terminal does not perform any processing.

[0159] In a possible embodiment, the socket management class SocketManager also defines method ③ the initialization method init(), which is responsible for initializing the member variables socket, input stream, and output stream; method ④ the connection closing method closeConnection(), which is used to disconnect the socket connection, thereby disconnecting from the device terminal.

[0160] In a possible embodiment, after receiving the device operation result, the device operation result is announced by voice, including successful operation, or including operation failure and the reason for failure.

[0161] Please refer to Figure 9 , Figure 9 which is a class diagram of a control terminal shown in an exemplary embodiment of the present application, as Figure 9As shown in the figure, in the control terminal, the direct connection management class includes an initialization method for instantiating the direct connection management class, a connection method for passing in a channel instance during connection, and a group owner search method for searching for devices; the channel class includes a closing method for closing the channel after the connection is closed; the broadcast receiver includes member variables of the direct connection management class and a device listener, as well as a group owner information callback function; the socket management class includes member variables of a socket, an input stream, and an output stream, and includes an initialization method, a listening method, a data transmission method, and a connection closing method.

[0162] Step S820: Parse the text data into formatted data. The formatted data includes a request type identifier and request data. The request type identifier is determined according to the request type of the text data and a first mapping relationship, and the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type.

[0163] Among them, the first mapping relationship is pre-configured, and the relationship between the request type identifier and the request type can be a one-to-one relationship.

[0164] In addition, the request type refers to the type of user intention represented by the voice control instruction.

[0165] In an embodiment, the request type is one of a page operation request, an application launch request, and a semantic search request; if the request type is a page operation request, the formatted data further includes an index parameter, and the index parameter is an identifier that uniquely identifies the target element referred to by the request data.

[0166] In this embodiment, when the request type is a page operation request, considering that there are multiple elements with the same keyword among the current page root element and its child elements, an index parameter is introduced. In this way, when there are multiple elements with the same keyword, it can play a role in identifying the target element corresponding to the request data, ensuring the reliability of subsequent execution by the device terminal.

[0167] Exemplarily, the text data is parsed into formatted data according to a preset parsing format. If the request type is a page operation request, the preset parsing format includes a request type identifier field, a request data field, and an index parameter field; if the request type is an application launch request or a semantic search request, the preset parsing format includes a request type identifier field and a request data field.

[0168] Exemplarily, the preset parsing format is a Json string.

[0169] As a possible embodiment, three types of semantics are supported: ① Keywords, suitable for the scenario of "what you see is what you say", that is, the text that the user sees on the device interface. For example, "Start playing". The third party will parse the user's semantics and return it as a Json string, {"action":"word","data":"Start playing","index":0}; ② Open a certain application, which is a fixed statement. For example, open a certain music software. It will be parsed into a Json string according to "open" and returned, {"action":"opeanApp","data":"a certain music software"}; ③ What the user says casually. For example, "I want to listen to the songs of a certain (singer's name)", which is parsed into a Json string and returned, {"action":"semantic ","data":"music: / / artist?name=a certain (singer's name)"}. After parsing the formatted data, it is further encapsulated into a voice control instruction.

[0170] Step S830, encapsulate the formatted data into a voice control instruction, and the voice control instruction carries a request type identifier and request data.

[0171] Exemplarily, according to a preset encapsulation format, the formatted data is encapsulated into a voice control instruction. If the request type is a page operation request, the preset encapsulation format includes a request type identifier field, a request data field, and an index parameter field; if the request type is an application start request or a semantic search request, the preset parsing format includes a request type identifier field and a request data field.

[0172] Exemplarily, regardless of which one of the page operation request, application start request, and semantic search request the request type is, the preset encapsulation format further includes a communication identifier field.

[0173] In this way, it can identify a communication between the device terminal and the control terminal, facilitate tracking the communication connection process between the two and the flow path of the interaction information, monitor the entire life cycle of the communication, and facilitate problem troubleshooting and log analysis.

[0174] Exemplarily, if the request type is a page operation request, taking the voice control to click on the page element "OK" as an example, the generated voice control instruction is as follows:

[0175] {

[0176] "requestId":"a1d16b58083347d19740198f43d7ee22",

[0177] "action":"word",

[0178] "data":"OK",

[0179] "index":0

[0180] }

[0181] Among them, "requestId":"a1d16b58083347d19740198f43d7ee22" is the communication identification field, which can be a 32-bit unique identifier generated by the UUID (Universally Unique Identifier) utility class provided by Java; "action":"word" is the request type identification field; "data":"determine" is the request data field, that is, the element that can be seen on the device terminal screen as said by the user; "index":0 is the index identification field, and the value of this field depends on what the user says. For example: when the user says "the second determine", the value is 1; when the user says "the first determine", the value is 0; when the user does not specify the nth determine, it defaults to the first determine, that is, the index defaults to 0.

[0182] Exemplarily, if the request type is an application opening request, taking the voice control to open a certain music software as an example, the encapsulated voice control instruction is as follows:

[0183] {

[0184] "requestId":"de06b23e5ea9408785fe76b87d64e72",

[0185] "action":"openApp",

[0186] "data":"a certain music software"

[0187] }

[0188] Among them: "requestId":"de06b23e5ea9408785fe76b87d64e72" is the communication identification field, which can be a 32-bit unique identifier generated by the UUID utility class provided by Java; "action":"openApp" is the request type identification field; "data":"a certain music software" is the request data field, that is, the name of the application that the user hopes to open.

[0189] Exemplarily, if the request type is a semantic search request, taking listening to the songs of a certain (singer's name) as an example, the encapsulated voice control instruction is as follows:

[0190] {

[0191] "requestId":6fc07c6c60704212bc1af04870130196",

[0192] "action":"semantic",

[0193] "data":"music: / / artist?name=某某(singer's name)"

[0194] }

[0195] Among them: "requestId":6fc07c6c60704212bc1af04870130196" is the request type identification field, which can be a 32-bit unique identifier generated by the UUID tool class provided by Java; "action":"semantic" is the request type identification field; "data":"music: / / artist?name=某某(singer's name)" is the request data field, that is, the device activity operation desired by the user.

[0196] In addition, the request data field in the format of "data":"music: / / artist?name=某某(singer's name)" is only explained using music as an example. In the actual implementation process, it can be adapted according to the URL Scheme (Uniform Resource Locator Scheme) supported by various applications on the market. Each application developer center will have a URL Scheme description to configure the request data field.

[0197] Step S840: Send a voice control instruction to the device terminal so that the device terminal operates the electronic device according to the voice control instruction.

[0198] At this point, the control terminal completes sending the voice control command and performs voice control on the electronic device.

[0199] In the above-mentioned electronic device control method, after the control terminal establishes a communication connection with the device terminal, it starts to capture user audio data and convert it into text data, and then parses the text data into formatted data, the formatted data includes a request type identifier and request data, and then encapsulates the formatted data into a voice control instruction, the voice control instruction carries the request type identifier and request data, and finally sends the voice control instruction to the device terminal, so that the device terminal operates the electronic device according to the voice control instruction to realize voice control of the electronic device, provides support for different user intentions, that is, diverse user voices, generates voice control instructions of different request types, and also facilitates classification processing by the device terminal, so as to meet the precise execution of different voice control requests. In this way, through a microphone-free voice adaptation solution, voice control of electronic devices can be realized without relying on a microphone, which not only reduces costs but also improves user experience.

[0200] See also Figure 10 , Figure 10 1 is a flowchart of a specific electronic device control method shown in an exemplary embodiment of the present application. The specific electronic device control method includes establishing a communication connection between a device terminal and a control terminal, transmitting data, and responding to a voice control command from the device terminal. Figure 10 As shown, taking the electronic device as a vehicle system and the device terminal as a vehicle-mounted terminal as an example, the specific electronic device control method is described in detail as follows:

[0201] The first step is to establish a communication connection. The direct connection management classes of the vehicle side and the control terminal are initialized respectively. In the vehicle side, a direct connection group is created through the direct connection management class after initialization, and the control terminal is waiting for connection. In the control terminal, a direct connection instance is created through the direct connection management class after initialization, and a broadcast receiver is created and initialized. Then the control terminal searches for the group owner through the direct connection management class, and calls back after searching the vehicle side. The broadcast receiver displays it to the user. The user clicks on the vehicle side device to connect, triggering the connection event of the control terminal and accessing the direct connection group of the vehicle side. If the connection is successful, the control terminal calls back the host information of the vehicle computer, and the vehicle side waits for the control terminal to send information. At this point, the vehicle side and the control terminal complete the communication connection.

[0202] Then comes the data transmission. The control terminal captures the audio data input by the user through the socket management class, converts it into text data, parses and encapsulates it, obtains the voice control command, and sends the voice control command to the vehicle end. The voice control command carries the request type identifier and request data.

[0203] Next is the vehicle-side response to the voice control command. The vehicle-side service management class determines the request type according to the request type identifier, and selects the corresponding processing class from the page operation processing class, application start processing class, and semantic search processing class to respond to the request data and execute the vehicle-side operation.

[0204] Finally, there is also data transmission. That is, after the vehicle terminal finishes execution, it sends the vehicle machine operation result to the control terminal, and the control terminal receives the vehicle machine operation result and performs voice broadcast.

[0205] Please refer to Figure 11 , Figure 11 which is a block diagram of an electronic device control system shown in an exemplary embodiment of the present application. This system can be applied to Figure 1 or Figure 2 the implementation environments shown. It should be understood that this system can also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environments applicable to this system.

[0206] As Figure 11 shown, in an exemplary embodiment, the electronic device control system 1100 at least includes a device terminal 1110 and a control terminal 1120, which are introduced in detail as follows:

[0207] The device terminal 1110 is used to create a direct connection group, and the device terminal is independent of the electronic device or integrated in the electronic device;

[0208] The control terminal 1120 is used to establish a communication connection with the device terminal 1110 according to the direct connection group; if the communication connection is successfully established, it captures user audio data and converts the user audio data into text data; it parses the text data into formatted data, and the formatted data includes a request type identifier and request data, and the request type identifier is determined according to the request type of the text data and the first mapping relationship, and the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type; it encapsulates the formatted data into a voice control instruction, and the voice control instruction carries the request type identifier and the request data; it sends the voice control instruction to the device terminal;

[0209] The device terminal 1110 is also used to receive the voice control instruction from the control terminal 1120; according to the request type identifier in the voice control instruction and the first mapping relationship, it determines the target request type of the voice control instruction; according to the target request type and the preset second mapping relationship, it determines the target processing class of the voice control instruction, and the second mapping relationship is a one-to-one correspondence between the request type and the processing class; it calls the target processing class and operates the electronic device according to the request data in the voice control instruction.

[0210] It should be noted that the electronic device control system provided in the above embodiment and the electronic device control methods applied to the device terminal and the control terminal provided in the above embodiment belong to the same concept. The content of the operations performed by each module has been described in detail in the method embodiment, and will not be repeated here.

[0211] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0212] The units described in the embodiments of the present application can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation to the units themselves in some cases.

[0213] The above embodiments are only used to exemplarily illustrate the principles and effects of the present application, rather than to limit the present application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present application should still be covered by the claims of the present application.

Claims

1. A method for controlling an electronic device, characterized in that: Applied to a device terminal, the device terminal is independent of the electronic device or integrated in the electronic device, the method comprises: Receiving a voice control instruction from a control terminal, the device terminal establishing a communication connection with the control terminal by creating a direct connection group, the voice control instruction carrying a request type identifier and request data; Determining a target request type of the voice control instruction according to the request type identifier and a preset first mapping relationship, wherein the first mapping relationship is a one-to-one correspondence between the request type identifier and the request type; Determining a target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, wherein the second mapping relationship is a one-to-one correspondence between the request type and the processing class; Calling the target processing class and operating the electronic device according to the request data, including: if the target request type is a page operation request, obtaining a page information processing class, the page information processing class including an accessibility service class and a page root element processing class, the accessibility service class defining a root element acquisition method and implementing a page root element listening interface; calling the page root element processing class to create a root element listening instance, and calling the root element acquisition method to acquire the page root element, and calling back the page root element and assigning it to the root element listening instance; operating the electronic device according to the assigned root element listening instance and the request data.

2. The electronic device control method according to claim 1, characterized in that: The target request type is one of a page operation request, an application start request and a semantic search request; the target processing class is one of a page information processing class, an application start processing class and a semantic search processing class.

3. The electronic device control method according to claim 1, characterized in that: The step of operating the electronic device according to the assigned root element monitoring instance and the request data includes: Calling the search element method in the page root element processing class, and traversing the page root element and the sub-elements corresponding to each of the page root elements in the search element method; If there is a target element matching the request data, a click event of the target element is triggered to operate the electronic device.

4. The electronic device control method according to claim 3, characterized in that: The method for determining the target element includes: Comparing an index parameter with a preset parameter threshold, wherein the index parameter is carried in the voice control instruction, and the index parameter is an identifier that uniquely identifies a target element indicated by the request data; If the index parameter is greater than the parameter threshold, then in the process of traversing the page root element and the child elements, the number of times the element matching the request data is traversed is recorded, and when the number matches the index parameter, the element corresponding to the number is determined as the target element.

5. The electronic device control method according to claim 1, characterized in that: The calling of the target processing class and operating the electronic device according to the request data includes: If the target request type is a semantic search request, obtaining a semantic search processing class; Call the semantic search processing class to obtain the data parsing method and activity starting method; Parsing the request data using the data parsing method to obtain an activity instance, wherein the activity instance includes a target application package name and an activity name; Passing the activity instance to the activity start method triggers an activity start event to operate the electronic device.

6. The electronic device control method according to any one of claims 1 to 5, characterized in that: Before receiving the voice control instruction from the control terminal, the method further includes: Detecting whether the electronic device has a voice capture and recognition function; If the electronic device does not have the voice capture and recognition function, sending a voice control command through the control terminal to control the electronic device; If the electronic device has the voice capture and recognition function, it is detected whether the voice capture and recognition function has a fault, and if the voice capture and recognition function has a fault, a voice control instruction is sent through the control terminal to control the electronic device.

7. A method for controlling an electronic device, characterized in that: The electronic device control method according to any one of claims 1 to 6 is applied to a control terminal, the method comprising: If the communication connection with the device terminal is successfully established, capturing user audio data and converting the user audio data into text data, wherein the communication connection is established through a direct connection group created by the device terminal, and the device terminal is independent of the electronic device or integrated in the electronic device; Parsing the text data into formatted data, the formatted data including a request type identifier and request data, the request type identifier being determined according to a request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; Encapsulating the formatted data into a voice control instruction, wherein the voice control instruction carries the request type identifier and the request data; The voice control instruction is sent to the device terminal so that the device terminal operates the electronic device according to the voice control instruction.

8. The electronic device control method according to claim 7, characterized in that: The request type is one of a page operation request, an application start request, and a semantic search request; If the request type is a page operation request, the formatted data further includes an index parameter, where the index parameter is an identifier that uniquely identifies the target element pointed to by the request data.

9. An electronic equipment control system, characterized in that: The system comprises: A device terminal, used for creating a direct connection group, wherein the device terminal is independent of the electronic device or integrated in the electronic device; A control terminal, configured to establish a communication connection with the device terminal according to the direct connection group; if the communication connection is successfully established, capture user audio data and convert the user audio data into text data; parse the text data into formatted data, the formatted data including a request type identifier and request data, the request type identifier being determined according to a request type of the text data and a first mapping relationship, the first mapping relationship being a one-to-one correspondence between the request type identifier and the request type; encapsulate the formatted data into a voice control instruction, the voice control instruction carrying the request type identifier and the request data; and send the voice control instruction to the device terminal; The device terminal is also used to receive the voice control instruction from the control terminal; determine the target request type of the voice control instruction according to the request type identifier in the voice control instruction and the first mapping relationship; determine the target processing class of the voice control instruction according to the target request type and a preset second mapping relationship, the second mapping relationship being a one-to-one correspondence between the request type and the processing class; call the target processing class to operate the electronic device according to the request data in the voice control instruction, including: if the target request type is a page operation request, obtain a page information processing class, the page information processing class includes an accessibility service class and a page root element processing class, the accessibility service class defines a root element acquisition method and implements a page root element listening interface; call the page root element processing class to create a root element listening instance, and call the root element acquisition method to obtain the page root element, and call back the page root element and assign it to the root element listening instance; operate the electronic device according to the assigned root element listening instance and the request data.

Images