A device operation method and system based on voice

Abstract

The present invention provides a device operation method and system based on voice, the method comprising: converting voice data of a user to be processed for operating an intelligent electronic device into a voice instruction to be processed; when there is a preset instruction template, matching the voice instruction to be processed with the instruction template; if the target instruction template cannot be matched, or the instruction template does not exist, determining the interaction path to be processed from a plurality of sample interaction paths determined based on a pre-built interface jump relationship diagram; and operating the intelligent electronic device according to the interaction path to be processed. The intelligent electronic device can be operated by voice operation to trigger the corresponding function, thereby reducing the difficulty of operating the intelligent electronic device and improving the efficiency of operating the intelligent electronic device.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a device operation method and system based on speech. Background Art

[0002] At present, when operating the interface of smart electronic devices such as mobile phones, tablet computers and televisions, the operation mainly relies on hand touch. However, when the user's hands are occupied, or the user is an elderly person, a person with a physical disability or a person with a visual impairment, it is very difficult to operate the smart electronic device by hand touch and the operation efficiency is low. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a device operation method and system based on voice to solve the problems of high operating difficulty and low operating efficiency in the current way of operating smart electronic devices.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] A first aspect of an embodiment of the present invention discloses a voice-based device operation method, the method comprising:

[0006] Analyze the voice data of the user to be processed for operating the intelligent electronic device to obtain the corresponding voice command to be processed;

[0007] When there is at least one preset instruction template, the voice instruction to be processed is matched using the at least one instruction template, each instruction template has a corresponding predetermined sample interaction path, and each sample interaction path is operation-related information for triggering a corresponding function of the intelligent electronic device;

[0008] If the target instruction template is not matched from the at least one instruction template, or when the at least one instruction template does not exist, the following steps are performed to determine and obtain the interaction path to be processed;

[0009] operating the intelligent electronic device according to the interaction path to be processed;

[0010] The following steps include:

[0011] Calculating the similarity between the voice instruction to be processed and each of the sample interaction paths, and determining at least one candidate interaction path from each of the sample interaction paths based on the similarity;

[0012] constructing a decision tree using the at least one candidate interaction path;

[0013] Based on the decision tree, an inquiry message is played to the user to be processed, voice reply messages of the user to be processed are collected in real time, and the interaction path to be processed is determined from the at least one candidate interaction path through the decision tree and the voice reply messages collected in real time.

[0014] Preferably, the method further comprises:

[0015] If a target instruction template corresponding to the voice instruction to be processed is obtained by matching from the at least one instruction template, the sample interaction path corresponding to the target instruction template is used as the interaction path to be processed of the voice instruction to be processed.

[0016] Preferably, the instruction template is generated by historical voice instructions of a first sample user for operating a corresponding intelligent electronic device, and the first sample user includes or does not include the user to be processed.

[0017] Preferably, the sample interaction path is determined based on a pre-constructed interface jump relationship diagram, and the interface jump relationship diagram is constructed based on pre-collected operation-related information of the smart electronic device corresponding to the second sample user operation, and the second sample user includes or does not include the user to be processed.

[0018] Preferably, the process of determining the sample interaction path includes:

[0019] Based on the pre-collected operation-related information of the intelligent electronic device corresponding to the second sample user operation, an operation list is constructed, wherein each operation-related information in the operation list at least includes: a page and a page layout corresponding to the operation-related information, wherein the page layout at least includes elements in the corresponding page;

[0020] Clustering the pages in the operation list to obtain corresponding page clustering results;

[0021] Based on the page clustering result, clustering the elements contained in each type of page in the operation list to obtain corresponding element clustering results;

[0022] Using the page clustering results and the element clustering results, an interface jump relationship graph is constructed, wherein each node in the interface jump relationship graph is an element in the operation list, each node in the interface jump relationship graph has at least one edge pointing from other nodes to itself, and each node corresponding to each element in the operation list has at least one edge with itself as the starting node;

[0023] Based on the interface jump relationship diagram and the page clustering result, combined with the information indicating the repeated occurrence of pages in the operation list, the operation list is segmented to obtain a plurality of sample interaction paths.

[0024] Preferably, after obtaining the multiple sample interaction paths, the method further includes:

[0025] For each sample interaction path, the input parameter corresponding to the sample interaction path, the source of the input parameter and the property of the input parameter are determined, and the property of the input parameter is used to indicate that the input parameter is a list selection parameter or a text input parameter.

[0026] Preferably, calculating the similarity between the voice instruction to be processed and each of the sample interaction paths, and determining at least one candidate interaction path from each of the sample interaction paths based on the similarity, comprises:

[0027] Determining the semantic vector of the voice command to be processed by using a preset pre-trained model;

[0028] Calculating a cosine value of a first angle between a semantic vector of the voice instruction to be processed and a semantic vector of each predetermined sample interaction path, and calculating a cosine value of a second angle between the semantic vector of the voice instruction to be processed and a semantic vector of each instruction template, wherein the semantic vector of the sample interaction path is predetermined and the semantic vector of the instruction template is predetermined;

[0029] Based on all the first angle cosine values ​​and the second angle cosine values, a standardized score of each of the first angle cosine values ​​is calculated to obtain the similarity between the voice instruction to be processed and each of the sample interaction paths, and a standardized score of each of the second angle cosine values ​​is calculated to obtain the similarity between the voice instruction to be processed and each of the instruction templates;

[0030] The sample interaction paths with the similarity greater than 0 are taken as candidate interaction paths, and the sample interaction paths corresponding to the instruction templates with the similarity greater than 0 are taken as candidate interaction paths.

[0031] Preferably, the sample interaction path includes at least a plurality of elements and their operation sequence, and the process of determining the semantic vector of the sample interaction path includes:

[0032] Extracting text information of each element in the sample interaction path and arranging them in operation order to obtain a keyword sequence;

[0033] splicing each keyword in the keyword sequence into a spliced ​​sentence;

[0034] Using a preset pre-trained model, the word vector of each keyword in the keyword sequence is calculated, all the word vectors are added together to obtain a word vector part, and the concatenated sentence is embedded to obtain a sentence vector part;

[0035] The word vector part and the sentence vector part are unitized and added in equal proportion to obtain the semantic vector of the sample interaction path.

[0036] Preferably, constructing a decision tree using the at least one candidate interaction path includes:

[0037] Construct a bag-of-words model for each candidate interaction path;

[0038] Extracting non-parameter texts from all the candidate interaction paths and constructing a text set, each element in the text set corresponds to a dimension in the vector space of the bag-of-words model;

[0039] Taking each of the candidate interaction paths as a sample, taking the similarity between each of the candidate interaction paths and the voice instruction to be processed as a weight, and taking the vector representation of the bag-of-words model of each of the candidate interaction paths as a feature, a weighted decision tree is constructed, wherein each leaf node in the decision tree corresponds to one of the candidate interaction paths, and each internal node in the decision tree corresponds to one of the non-parametric texts.

[0040] Preferably, the step of playing an inquiry message to the user to be processed based on the decision tree, collecting voice reply messages of the user to be processed in real time, and determining the interaction path to be processed from the at least one candidate interaction path through the decision tree and the voice reply messages collected in real time includes:

[0041] Traversing from the root node of the decision tree, playing a query message to the user to be processed each time entering any internal node of the decision tree, and collecting voice reply messages of the user to be processed in real time;

[0042] Based on the voice reply message collected in real time, the next node of the decision tree to be entered is determined, and the step of playing the inquiry message to the user to be processed is repeated until any leaf node of the decision tree is entered, and the candidate interaction path corresponding to the leaf node entered is used as the interaction path to be processed.

[0043] Preferably, after determining the interaction path to be processed from the at least one candidate interaction path, the method further includes:

[0044] Playing the interaction path information corresponding to the interaction path to be processed to the user to be processed, and collecting the voice confirmation message of the user to be processed in real time;

[0045] If the voice confirmation message indicates that the to-be-processed user denies the to-be-processed interaction path, the to-be-processed interaction path is deleted from all the candidate interaction paths, and the process returns to the step of constructing a decision tree using the at least one candidate interaction path.

[0046] Preferably, operating the intelligent electronic device according to the interaction path to be processed includes:

[0047] Determine all operations and their order in the interaction path to be processed;

[0048] According to the operation sequence in the interaction path to be processed, the operations in the interaction path to be processed are executed on the graphic interface of the intelligent electronic device.

[0049] Preferably, after operating the intelligent electronic device according to the interaction path to be processed, the method further includes:

[0050] The function parameters in the voice instruction to be processed are deleted, and a command template is constructed based on the remaining part of the voice instruction to be processed.

[0051] A second aspect of an embodiment of the present invention discloses a voice-based device operating system, the system comprising:

[0052] The parsing unit is used to parse the voice data of the user to be processed for operating the intelligent electronic device to obtain the corresponding voice command to be processed;

[0053] A matching unit, configured to match the voice command to be processed by using at least one preset command template when there is at least one preset command template, each of the command templates having a corresponding predetermined sample interaction path, each of the sample interaction paths being operation-related information for triggering a corresponding function of the intelligent electronic device;

[0054] A processing unit, configured to, if the target instruction template is not matched from the at least one instruction template, or when the at least one instruction template does not exist, execute the following steps to determine the interaction path to be processed, and execute the operation unit;

[0055] The following steps include:

[0056] Calculating the similarity between the voice instruction to be processed and each of the sample interaction paths, and determining at least one candidate interaction path from each of the sample interaction paths based on the similarity;

[0057] constructing a decision tree using the at least one candidate interaction path;

[0058] Playing an inquiry message to the user to be processed based on the decision tree, collecting voice reply messages of the user to be processed in real time, and determining the interaction path to be processed from the at least one candidate interaction path through the decision tree and the voice reply messages collected in real time;

[0059] The operating unit is used to operate the intelligent electronic device according to the interaction path to be processed.

[0060] Based on the above-mentioned embodiment of the present invention, a device operation method and system based on voice is provided. The method is as follows: converting the voice data of the user to be processed for operating the intelligent electronic device into the voice command to be processed; when there is a preset command template, using the command template to match the voice command to be processed; if the target command template cannot be matched, or the command template does not exist, determining the interaction path to be processed from multiple sample interaction paths determined based on the pre-built interface jump relationship diagram; operating the intelligent electronic device according to the interaction path to be processed. It is realized that the intelligent electronic device is operated in a voice operation mode to trigger the corresponding function, reducing the difficulty of operating the intelligent electronic device and improving the efficiency of operating the intelligent electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0062] Figure 1 A flow chart of a method for operating a device based on voice provided by an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of determining an interaction path to be processed from candidate interaction paths provided by an embodiment of the present invention;

[0064] Figure 3 A flow chart for determining a sample interaction path provided by an embodiment of the present invention;

[0065] Figure 4 A flowchart for determining a candidate interaction path provided by an embodiment of the present invention;

[0066] Figure 5 A flowchart of constructing a decision tree provided by an embodiment of the present invention;

[0067] Figure 6 A schematic diagram of each dimension of the bag-of-words model provided by an embodiment of the present invention;

[0068] Figure 7A structural block diagram of a voice-based device operating system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0071] As can be seen from the background technology, currently, when operating on the interface of a smart electronic device, the main method is to use hand touch to trigger a function of the smart electronic device. However, when the user's hands are occupied, or for users such as the elderly, people with physical disabilities or people with visual impairments, it is difficult to operate the smart electronic device by hand touch and the operation efficiency is low.

[0072] Therefore, an embodiment of the present invention provides a voice-based device operation method and system, which converts the voice data of the user to be processed for operating the smart electronic device into a voice instruction to be processed. When there is a preset instruction template, the instruction template is used to match the voice instruction to be processed. If the target instruction template cannot be matched, or the instruction template does not exist, the interaction path to be processed is determined from multiple sample interaction paths determined based on a pre-built interface jump relationship diagram. The smart electronic device is operated according to the determined interaction path to be processed, and the relevant functions of the smart electronic device are triggered by voice operation, so as to reduce the difficulty of operating the smart electronic device and improve the efficiency of operating the smart electronic device.

[0073] It should be noted that, by using a device operation method and system based on voice provided by an embodiment of the present invention, the intelligent electronic device can be operated by voice operation to trigger any function in the graphical interface of the intelligent electronic device. For example, when the user says "Open Meituan shared bicycle 24637xxx62", the device operation method based on voice provided by an embodiment of the present invention simulates the user operation in the graphical interface of the intelligent electronic device (open Meituan - click to ride - click to scan the code to use the bicycle - click to enter the bicycle number - enter 24637xxx62 in the text box - click to confirm), and completes the unlocking of the shared bicycle. See the following embodiments for details.

[0074] It should be further explained that the interactive path involved in the embodiments of the present invention specifically refers to the operation-related information corresponding to the user operating the intelligent electronic device to trigger a certain function. The interactive path for triggering a certain function at least includes: each step of the user's operation and its type, the order of each step of the operation, the object of each step of the operation (element in the graphical interface), whether each step of the operation carries parameters, the parameter type corresponding to the operation carrying parameters (list selection parameters or text input parameters, etc.), the page layout and page screenshots of the graphical interface before and after each step of the operation, etc. In other words, each interactive path corresponds to a function of the intelligent electronic device, and the user can trigger a function of the intelligent electronic device through a voice command.

[0075] See also Figure 1 , shows a flow chart of a voice-based device operation method provided by an embodiment of the present invention, the device operation method comprising:

[0076] Step S101: parsing the voice data of the user to be processed for operating the intelligent electronic device to obtain the corresponding voice command to be processed.

[0077] It should be noted that, in the intelligent electronic device operated by the user to be processed, necessary data is downloaded from the server in advance, and the necessary data includes at least: the sample interaction path mentioned in the following steps and embodiments, the attributes of the sample interaction path (required parameters, the source of the parameters on the graphical interface and the properties of the parameters, etc.) and the semantic vector of the sample interaction path and other data.

[0078] In the specific implementation of step S101, voice data of a user to be processed for operating an intelligent electronic device (ie, triggering a certain function through the voice data) is obtained, and the voice data is parsed to obtain a corresponding voice instruction to be processed.

[0079] It should be noted that when the first sample user uses the corresponding intelligent electronic device, at least one instruction template is generated according to the historical voice instructions of the first sample user for operating the corresponding intelligent electronic device, each instruction template has a corresponding predetermined sample interaction path, and the first sample user includes or does not include the user to be processed. In other words, the historical voice instructions of the first sample user (including the user to be processed and / or other users) for operating the corresponding intelligent electronic device are collected in advance, and an instruction template (the template type can be a regular expression template) is constructed according to the collected historical voice instructions.

[0080] It should be further explained that, in actual applications, there may be instruction templates, or there may not be instruction templates (ie, no user has used a voice-based device operation method provided by an embodiment of the present invention to operate an intelligent electronic device).

[0081] When there is at least one instruction template, the following step S102 is executed.

[0082] When at least one instruction template does not exist, the following step S104 is directly executed, that is, when there is no instruction template, there is no need to execute the template matching process (ie, step S102 and step S103).

[0083] Each sample interaction path is operation-related information for triggering a corresponding function (the function corresponding to the sample interaction path) of a smart electronic device. The sample interaction path is determined based on a pre-constructed interface jump relationship diagram. The interface jump relationship diagram is constructed based on pre-collected operation-related information of the smart electronic device corresponding to the second sample user operation. The second sample user includes or does not include the user to be processed.

[0084] That is to say, the operation-related information corresponding to each operation performed by the second sample user (including multiple users, including the user to be processed) on the smart electronic device is collected in advance. For example, when the second sample user performs any one or more operations such as clicking the screen, clicking the mouse, tapping the keyboard, pressing a button, making a gesture, and giving a voice command on the smart electronic device, the operation-related information corresponding to each operation is recorded. The operation-related information includes but is not limited to: specific information of the operation (including the operation object, etc.), page screenshots before and after the operation, and page layout before and after the operation; the page layout includes but is not limited to: all elements in the page (such as buttons, lists, labels, text boxes and menus, etc.), and the parallel and inclusion relationships between elements; wherein each element contains its own attributes (such as type, text, size, supported operations, and position on the interface, etc.).

[0085] Step S102: When there is at least one preset instruction template, the at least one preset instruction template is used to match the voice instruction to be processed.

[0086] In the specific implementation of step S102, when there is at least one preset instruction template, the rule of regular expression is used to determine whether the instruction to be processed matches any instruction template. If the instruction to be processed can match any instruction template, step S103 is executed; if the instruction to be processed fails to match any instruction template, step S104 is executed.

[0087] It is understandable that the instruction template can be accumulated with the interaction between the first sample user and the corresponding intelligent electronic device. At the same time, when the instruction template is generated, the generated instruction template is mapped to the preset sample interaction path, that is, each instruction template has a corresponding sample interaction path.

[0088] Step S103: If a target instruction template corresponding to the voice instruction to be processed is matched from at least one instruction template, the sample interaction path corresponding to the target instruction template is used as the interaction path to be processed of the voice instruction to be processed, and step S107 is executed.

[0089] It can be seen from the content of the above step S102 that each instruction template has a corresponding sample interaction path. In the process of implementing step S103, if the target instruction template corresponding to the voice instruction to be processed can be matched from the instruction template, the sample interaction path corresponding to the target instruction template is used as the interaction path to be processed for the voice instruction to be processed. Since the sample interaction path corresponding to the target instruction template has triggered a function of the intelligent electronic device, it means that the intention and parameters of the sample interaction path corresponding to the target instruction template have been determined. Therefore, after using the sample interaction path corresponding to the target instruction template as the interaction path to be processed for the voice instruction to be processed, step S107 is directly executed to operate the intelligent electronic device according to the interaction path to be processed.

[0090] For example: assuming that the intention of the sample interaction path corresponding to the target instruction template is "send a WeChat red envelope", and the parameters of the sample interaction path are "send the red envelope to A" and "the amount of the red envelope is 5 yuan", then the intention of the determined interaction path to be processed is also "send a WeChat red envelope", and the parameters of the interaction path to be processed are "send the red envelope to A" and "the amount of the red envelope is 5 yuan". At this time, the corresponding function can be completed by directly operating the smart electronic device according to the interaction path to be processed.

[0091] Step S104: If the target instruction template is not matched from at least one instruction template, or when at least one instruction template does not exist, calculate the similarity between the voice instruction to be processed and each sample interaction path, and determine at least one candidate interaction path from each sample interaction path based on the similarity, and execute steps S105 to S106.

[0092] In the process of implementing step S104, if a target instruction template matching the voice instruction to be processed cannot be determined from all instruction templates, or when there is no instruction template, the semantic vector of the voice instruction to be processed is calculated, and based on the semantic vector of the voice instruction to be processed, the semantic vector of the instruction template (predetermined) and the semantic vector of the sample interaction path (predetermined), the similarity between the voice instruction to be processed and each sample interaction path is calculated.

[0093] Based on the calculated similarity, at least one candidate interaction path is determined from each sample interaction path. Combined with the above content, it can be seen that each candidate interaction path also corresponds to a function of the intelligent electronic device, and the function corresponding to the candidate interaction path can also be called a candidate function.

[0094] Step S105: construct a decision tree using at least one candidate interaction path.

[0095] In the specific implementation of step S105, after the candidate interaction paths are determined from the sample interaction paths, a bag-of-words model corresponding to each candidate interaction path is constructed, that is, a bag-of-words model corresponding to each candidate interaction path is constructed.

[0096] A decision tree is constructed using the bag-of-words model of each candidate interaction path, the non-parameter text of each candidate interaction path, and the similarity between each candidate interaction path and the voice command to be processed. Each internal node of the decision tree corresponds to a non-parameter text, and each leaf node of the decision tree corresponds to a candidate interaction path.

[0097] Step S106: Based on the decision tree, an inquiry message is played to the user to be processed, and voice reply messages of the user to be processed are collected in real time. The interaction path to be processed is determined from at least one candidate interaction path through the decision tree and the voice reply messages collected in real time, and step S107 is executed.

[0098] In the process of specifically implementing step S106, traversal is performed starting from the root node of the decision tree, and an inquiry message is played to the user to be processed each time any internal node of the decision tree is entered. As described above, each internal node corresponds to a non-parameter text. Each time any internal node of the decision tree is entered, an inquiry message is played to the user to be processed, asking the user to be processed whether the non-parameter text corresponding to the internal node entered will appear during the function execution process, for example: asking the user to be processed "Do you need xx operation?"; the voice reply message of the user to be processed is collected in real time, and the next node of the decision tree to be entered is determined according to the voice reply message of the user to be processed, for example: if the user answers that the non-parameter text corresponding to the internal node entered does not appear, then the node on the left side of the internal node is entered; if the user answers that the non-parameter text corresponding to the internal node entered appears, then the node on the right side of the internal node is entered; the step of playing the inquiry message to the user to be processed is repeated until any leaf node of the decision tree is entered, and the candidate interaction path corresponding to the leaf node entered is used as the interaction path to be processed.

[0099] It is understandable that when an inquiry message is played to the user to be processed after entering a certain internal node of the decision tree, the user to be processed may answer "I don't know (or unclear words such as unclear)". At this time, the dimension corresponding to the current internal node entered is deleted from the vector space of the bag-of-words model, and the decision tree is rebuilt and the above steps are performed again until the interaction path to be processed is determined.

[0100] That is, the inquiry message is played to the user to be processed and the user's voice reply message is collected in real time to interact with the user to be processed, and then the interaction path to be processed corresponding to the voice instruction to be processed of the user to be processed is determined.

[0101] To better explain how to interact with the user to be processed, Figure 2 The schematic diagram of determining the interaction path to be processed from the candidate interaction paths is shown for illustration.

[0102] Need to be explained Figure 2 0-6 in the dialog is the order of the interaction with the user to be processed.

[0103] Sequence number 0 is the pending user's pending voice command, and its content is: Open the payment code:; At this time, the content of sequence number 1 is played to the pending user, and the content of sequence number 1 is: Do you need to click the "more function button"? The voice reply message of the pending user collected in real time is the content of sequence number 2, and the content of sequence number 2 is: Yes; The content of sequence number 3 is played to the pending user, and the content of sequence number 3 is: Do you need to click "start a group chat"? The voice reply message of the pending user collected in real time is the content of sequence number 4, and the content of sequence number 4 is: No; The content of sequence number 5 is played to the pending user, and the content of sequence number 5 is: Do you need to click "QR code payment"? The voice reply message of the pending user collected in real time is the content of sequence number 6, and the content of sequence number 6 is: Yes. Through the above 0-6 dialogue sequence, interact with the pending user, and finally determine the interaction path to be processed.

[0104] Preferably, after the interaction path to be processed is determined, that is, the function to be executed that the user to be processed wants to trigger (ie, the function corresponding to the interaction path to be processed) is determined, the function to be executed needs to be described to the user to be processed for further verification.

[0105] The specific verification method is: play the interaction path information corresponding to the interaction path to be processed to the user to be processed, and collect the voice confirmation message of the user to be processed in real time. If the voice confirmation message indicates that the user to be processed denies the interaction path to be processed, delete the interaction path to be processed from all candidate interaction paths, and return to the step of building a decision tree using at least one candidate interaction path. If the voice confirmation message indicates that the user to be processed confirms the interaction path to be processed, continue to execute the subsequent steps. During this process, the user to be processed can also end the voice interaction process.

[0106] It can be understood that, in the process of playing the interaction path information corresponding to the interaction path to be processed to the user to be processed for further verification, if the function corresponding to the interaction path to be processed has been triggered before (which can be determined based on the historical interaction process with the user to be processed or other users), the corresponding interaction path information will be played with the voice command used by the function corresponding to the interaction path to be processed that has been triggered, for example: playing to the user to be processed "Do you want to perform an operation of type 'xxxx'"; at this time, in order to protect the privacy of the user to be processed, the parameters in the played content can be replaced with specified parameters, for example: assuming that the function corresponding to the interaction path to be processed is "Send a 10 yuan red envelope to Xiao Ming", and the user to be processed or other users have triggered the function of "Send a 5 yuan red envelope to Xiao Hong", at this time, playing to the user to be processed "Do you want to perform an operation similar to 'Send a 5 yuan red envelope to Xiao Hong'" can be played to the user to be processed.

[0107] If the function corresponding to the interaction path to be processed has never been triggered, then the specific execution process of the function corresponding to the interaction path to be processed is described step by step to the user to be processed; for non-parameter operations, the type of operation and the text corresponding to the operation are directly described, and for operations with parameters, the type of operation is described. For example: for the function of sending a red envelope, the description is "Do you need to: open the red envelope, select an item in the list, click the more functions button, click the red envelope, enter text in the text box, and finally click to put money into the red envelope."

[0108] Step S107: operating the intelligent electronic device according to the interaction path to be processed.

[0109] In the process of implementing step S107, all operations and operation sequences in the interaction path to be processed are determined, and the operations in the interaction path to be processed are gradually executed on the graphical interface of the intelligent electronic device according to the operation sequence in the interaction path to be processed, so as to gradually simulate each operation of the user to be processed on the graphical interface of the intelligent electronic device until all operations in the interaction path to be processed are executed, and finally the function corresponding to the interaction path is triggered. In the process of executing all operations in the interaction path to be processed, for any operation, the type of the operation, the element (i.e., the object) of the operation, the parameters carried by the operation (or no parameters) and the parameter types thereof are used to complete the execution of the operation.

[0110] It can be understood that for any operation in the interaction path to be processed, if the operation does not carry parameters, an element to be operated is determined in the graphical interface of the intelligent electronic device based on the attributes of the element corresponding to the operation in the interaction path to be processed (such as text, position and size, etc.), and the simulation of the event is completed through the preset interface (i.e., the operation is executed).

[0111] It should be noted that, from the contents of the above steps, it can be known that the interaction path to be processed is: obtained by matching from the instruction template, or determined from various sample interaction paths.

[0112] For any operation in the interaction path to be processed, if the operation carries parameters, and if the interaction path to be processed is matched from the instruction template (the template will contain the corresponding parameter information), it means that the value of the parameter carried by the operation is known. At this time, directly select the list item corresponding to the operation or enter the corresponding text in the text box corresponding to the operation according to the value of the parameter to execute the operation. For example: assuming that the voice instruction to be processed is "Send a 30 yuan red envelope to Xiao Ming", and the target instruction template matched is "Send a xx yuan red envelope to xx", then it can be determined that the parameters corresponding to the operation carrying parameters in the interaction path to be processed are "Xiao Ming" and "30".

[0113] For any operation in the interaction path to be processed, if the operation carries a parameter, the value of the parameter of the operation is unknown and the parameter type is a list selection parameter (for example, selecting the object to send a red envelope on the WeChat homepage), the text matching is performed between the voice command to be processed and the elements of the graphical interface of the smart electronic device to determine the list item to be operated, and the preset interface is used to complete the simulation of the event (i.e., execute the operation). It is understandable that for smart electronic devices with small screens, it may not be possible to display all the contents of the list item. At this time, the list item can be scrolled to display more content of the list item.

[0114] For any operation in the interaction path to be processed, if the operation carries parameters, the value of the parameter of the operation is unknown and the parameter type is a text input parameter, the user to be processed is asked about the parameter content that needs to be input for the operation. Similarly, in the above content about the parameter type being a list selection parameter, if the list item that needs to be operated is not found, the parameter can also be determined by asking. It can be understood that the parameter inquiry to the user to be processed mainly includes two aspects, one is: the content of the previous operation, which is mainly used to help the user to be processed determine the specific information of the current page; the second is: the text around the control that requires the parameter (such as a text input box) is used as a prompt text to feed back to the user to be processed, indicating where the parameter needs to be used. For example: asking "after clicking the red envelope button" (the content of the previous operation) and asking "what do you need to enter in the single amount text box" (where the parameter needs to be used). The voice reply of the user to be processed to the inquiry is the value of the parameter. After determining the value of the parameter corresponding to the operation, according to the value of the parameter, the corresponding list item is selected or the corresponding text is entered in the text box, thereby completing the execution of the operation.

[0115] Through the above method, all operations in the interaction path to be processed are executed in the graphical interface of the intelligent electronic device, triggering the function corresponding to the interaction path to be processed.

[0116] Preferably, after successfully executing all operations in the interaction path to be processed, the voice instruction to be processed can be used to construct an instruction template. The specific method of constructing the instruction template is: delete the functional parameters in the voice instruction to be processed, and construct the instruction template based on the remaining part of the voice instruction to be processed.

[0117] It should be noted that for the interaction path to be processed (that is, the interaction path), it corresponds to a function. When executing each operation of the interaction path to be processed corresponding to the voice instruction to be processed, the variable part of the voice instruction to be processed is the function parameter. Delete the function parameters in the voice instruction to be processed, and the remaining part can be used to construct the instruction template. For example: assuming that the voice instruction to be processed is "send a red envelope of 10 yuan to Xiao Ming", its corresponding function parameters (that is, the variable part) are "Xiao Ming" and "10", and the function parameters of the voice instruction to be processed, the remaining part (that is, the instruction template) is "send a red envelope of <> yuan to <>", where "<>" means leaving it blank, which is the variable part in the instruction template.

[0118] Calculate the semantic vector of the instruction template (that is, the vector representation in the semantic space). The method of calculating the semantic vector of the instruction template is the same as the method of calculating the semantic vector of the sample interaction path. For details, see the following embodiment of the present invention Figure 3 The content shown in step S305 is understandable that after calculating the semantic vector of the instruction template, the blank part of the instruction template is replaced with preset parameters (for example, it can be replaced with the parameters used in the process of building the interface jump relationship diagram), so that the instruction template becomes a complete voice instruction.

[0119] The constructed instruction template and its semantic vector are stored locally in the smart electronic device and also uploaded to the server and shared among different users. This can achieve the accumulation of semantic data and the improvement of semantic understanding ability during the interaction between the user and the smart electronic device, and reduce the number of additional dialogue rounds required between the user and the smart electronic device.

[0120] It should be noted that in the process of operating an intelligent electronic device according to the interaction path to be processed, abnormal errors may occur, for example: the element to be operated cannot be found in the graphical interface of the intelligent electronic device, or a semantic understanding error occurs.

[0121] Preferably, when operating the intelligent electronic device according to the interaction path to be processed, if the element to be operated cannot be found, the shortest operation path is determined from the interface jump relationship diagram using a breadth-first search method, based on the nodes corresponding to all elements in the current graphical interface of the intelligent electronic device in the preset interface jump relationship diagram, with the goal of making the element to be operated reappear in the interface graphic of the intelligent electronic device. The intelligent electronic device is operated through the shortest operation path, so that the element to be operated reappears in the graphical interface of the intelligent electronic device. In this process, errors may also occur, and the aforementioned steps can be repeated several times. If an error still occurs, an error report is performed.

[0122] When operating an intelligent electronic device according to the interaction path to be processed, if a semantic understanding error occurs, the user to be processed is asked to reconfirm the parameters, for example, asking the user to be processed "Do you need to select xxx?".

[0123] In an embodiment of the present invention, the voice data of the user to be processed for operating the intelligent electronic device is converted into a voice instruction to be processed; when there is a preset instruction template, the instruction template is used to match the voice instruction to be processed; if the target instruction template corresponding to the voice instruction to be processed can be matched, the sample interaction path corresponding to the target instruction template is used as the interaction path to be processed of the voice instruction to be processed; if the target instruction template cannot be matched, or the instruction template does not exist, the interaction path to be processed is determined from multiple sample interaction paths determined based on a pre-built interface jump relationship diagram; the intelligent electronic device is operated according to the interaction path to be processed. It is realized that the intelligent electronic device is operated in a voice operation mode to trigger the corresponding function, which reduces the difficulty of operating the intelligent electronic device and improves the efficiency of operating the intelligent electronic device.

[0124] The above embodiments of the present invention Figure 1 For the process of determining the sample interaction path mentioned in step S104, see Figure 3 , shows a flow chart of determining a sample interaction path provided by an embodiment of the present invention, including:

[0125] Step S301: construct an operation list based on the pre-collected operation-related information of the intelligent electronic device corresponding to the second sample user operation.

[0126] It should be noted that each operation-related information in the operation list at least includes: a page corresponding to the operation-related information and a page layout, and the page layout at least includes elements in the corresponding page. For details, see the above-mentioned embodiment of the present invention. Figure 1 The content in step S104 is not repeated here.

[0127] In the specific implementation of step S301, within a certain period of time, operation-related information of the intelligent electronic device corresponding to the second sample user operation is collected, and an operation list is constructed based on the collected operation-related information, and the operation list is uploaded to the server.

[0128] Step S302: clustering the pages in the operation list to obtain corresponding page clustering results.

[0129] In the specific implementation of step S302, the server is used to cluster the pages included in the operation list to obtain a page clustering result, which indicates which type of page each page in the operation list belongs to.

[0130] It is understandable that in the process of clustering pages, a heuristic algorithm can be used to determine whether two pages are the same page based on important visual features on the page (such as the title, the tab bar and whether it is highlighted, the page border shape, etc.) and the functions supported by the page.

[0131] Step S303: Based on the page clustering result, cluster the elements contained in each type of page in the operation list to obtain corresponding element clustering results.

[0132] In the specific implementation of step S303 , based on the page clustering result, for each type of page in the operation list, all elements in the type of page are clustered to obtain a corresponding element clustering result.

[0133] It is understandable that in the process of clustering elements, a heuristic algorithm is used to determine whether two elements are the same element through the text, position, size of the elements in the page and the layout relationship between the elements.

[0134] Step S304: construct an interface jump relationship diagram using the page clustering results and the element clustering results.

[0135] It should be noted that each node in the interface jump relationship diagram is an element in the operation list. Each node in the interface jump relationship diagram has at least one edge pointing from other nodes to itself, and the node corresponding to each element in the operation list has at least one edge with itself as the starting node.

[0136] It should be further explained that each element in the operation list also contains the following information: which element on which page will be jumped to after operating a certain element on a certain page. In other words, there are at least two kinds of connections between elements, one is whether the two elements are the same (which can be determined based on the element clustering results), and the other is which elements will be reached by operating a certain element (which can be determined based on the operation list).

[0137] In the specific implementation of step S304, the page clustering results and the element clustering results are used to merge the pages of the same type and the elements of the same type in the operation list, thereby converting the operation list into an interface jump relationship diagram.

[0138] It can be understood that the interface jump relationship diagram is based on elements, that is, each node in the interface jump relationship diagram is an element. The edge in the interface jump relationship diagram starting from a certain node represents an operation on the node (that is, a class of elements). The edge starting from a certain node is also called the out-edge of the node. For example: node a1 has two out-edges, pointing to node b1 (a1 to b1) and node b2 (a1 to b2) respectively.

[0139] Each outgoing edge reaches a type of element, indicating that the operation corresponding to the outgoing edge can cause this type of element (the element reached) to appear on the graphical interface of the smart electronic device. Since a type of element can support multiple types of elements, that is, operating a type of element can cause multiple elements to appear on the graphical interface of the smart electronic device, each type of element has at least one or more outgoing edges. By using an interface jump relationship diagram based on elements, the jump relationship between pages can be better portrayed, as well as the state change relationship within the page (such as the text input state and the press-to-speak state in the WeChat chat page).

[0140] Step S305: Based on the interface jump relationship diagram and the page clustering result, combined with the information indicating the repeated occurrence of pages in the operation list, the operation list is segmented to obtain multiple sample interaction paths.

[0141] In the specific implementation of step S305, based on the interface jump relationship diagram and the page clustering results, and combined with the information indicating the repeated appearance of pages in the operation list, the operation list is segmented by the server to obtain multiple interaction fragments, wherein the beginning and the end of each interaction fragment are the same page. The continuous return operation (such as clicking the virtual return key) at the end of each interaction fragment is deleted to obtain multiple sample interaction paths.

[0142] It is understandable that for each page in the operation list, it is necessary to search for the same page forward. If the same page can be found, the path between the two same pages is segmented from the operation list, thereby obtaining the corresponding interaction fragment.

[0143] In some embodiments, when the operation list is split to obtain multiple interaction fragments, the multiple interaction fragments need to be merged or deleted. The specific merging method is: merge the interaction fragments that appear consecutively, for example: interaction fragment A and interaction fragment B appear consecutively (that is, if interaction fragment A appears, interaction fragment B will definitely appear), then merge interaction fragment A and interaction fragment B. Another example: interaction fragment A is followed by interaction fragment B (that is, the last page of interaction fragment A is exactly the first page of interaction fragment B), and the elements operated in the first page of interaction fragment B do not exist in the first page of interaction fragment A but exist in the last page of interaction fragment A (it should be noted here that the first page and the last page of interaction fragment A are the same page), then it means that interaction fragment B uses the result of interaction fragment A, and interaction fragment A and interaction fragment B need to be merged.

[0144] The specific deletion method is: if no parameter appears in an interaction fragment, the interaction fragment will be deleted.

[0145] After merging and deleting, multiple interaction fragments are obtained. The continuous return operations at the end of each interaction fragment are deleted to obtain multiple sample interaction paths. Each interaction fragment corresponds to a sample interaction path.

[0146] Preferably, after determining that multiple sample interaction paths are obtained, it is also necessary to determine the input parameters required for each sample interaction path. The specific method is: for each sample interaction path, determine the input parameters corresponding to the sample interaction path, the source of the input parameters (indicating on which specific page to complete the parameter input) and the nature of the input parameters. The nature of the input parameters is used to indicate whether the input parameters are list selection parameters or text input parameters.

[0147] It can be understood that by judging whether a text input operation is performed in the sample interaction path, if a text input operation is performed, the nature of the input parameter of the sample interaction path is a text input parameter. The nature of the input parameter of the sample interaction path can be determined to be a list selection parameter only when the list item selected by the sample interaction path meets one of the following three conditions: the first condition is: the list item contains multiple texts; the second condition is: the length of the text in the list item is greater than the specified length (for example, greater than 13 characters); the third condition is: the text in the list item is a named entity (such as a person's name, place name, organization name, number, time, etc.).

[0148] Preferably, the input parameters corresponding to each sample interaction path, the sources of the input parameters and the properties of the input parameters are stored in the server.

[0149] Preferably, after a plurality of sample interaction paths are determined, a semantic vector of each sample interaction path needs to be calculated, and the semantic vector of the sample interaction path is used as a vector representation of the function corresponding to the sample interaction path.

[0150] In some embodiments, the sample interaction path includes at least a plurality of elements and their operation order, and the method for calculating the semantic vector of the sample interaction path is as follows: extract the text information of each element in the sample interaction path and arrange them in the order of operation to obtain a keyword sequence. The keywords in the keyword sequence are spliced ​​into a spliced ​​sentence (without spaces) according to the operation order. Using a preset pre-trained model (such as a BERT pre-trained model, whose parameters can be 12 layers, 768 hidden layers and 110M), the word vectors of each keyword in the keyword sequence are calculated, all the word vectors are added to obtain the word vector part, and the spliced ​​sentence is embedded to obtain the sentence vector part; the word vector part and the sentence vector part are unitized and added in equal proportion to obtain the semantic vector of the sample interaction path. Among them, the word vector represents the keyword, and the sentence vector supplements the semantic relationship between the keywords.

[0151] In an embodiment of the present invention, an operation list is constructed using the operation-related information of the second sample user collected in advance, and the operation list is converted into an interface jump relationship diagram. The operation list is segmented according to the interface jump relationship diagram to obtain multiple sample interaction paths. The multiple sample interaction paths are used to determine the interaction path to be processed corresponding to the voice instruction to be processed, and the intelligent electronic device is operated according to the interaction path to be processed. The intelligent electronic device is operated in a voice operation mode to trigger the corresponding function, thereby reducing the difficulty of operating the intelligent electronic device and improving the efficiency of operating the intelligent electronic device.

[0152] The above embodiments of the present invention Figure 1 For the process of determining the candidate path involved in step S104, see Figure 4 , shows a flowchart of determining a candidate interaction path provided by an embodiment of the present invention, including the following steps:

[0153] Step S401: using a preset pre-trained model to determine a semantic vector of a voice instruction to be processed.

[0154] In the specific implementation of step S401, a preset pre-trained model (such as a BERT pre-trained model) is used to determine the semantic vector of the voice instruction to be processed. For details on how to determine the semantic vector of the voice instruction to be processed, please refer to the above-mentioned embodiment of the present invention. Figure 3 The content of calculating the semantic vector of the sample interaction path in step S305 will not be repeated here.

[0155] Step S402: Calculate the cosine value of the first angle between the semantic vector of the voice instruction to be processed and the semantic vector of each predetermined sample interaction path, and calculate the cosine value of the second angle between the semantic vector of the voice instruction to be processed and the semantic vector of each instruction template.

[0156] It should be noted that the semantic vector of the sample interaction path is predetermined, and the semantic vector of the instruction template is predetermined. The process of determining the semantic vector is detailed in the above embodiment of the present invention. Figure 3 The content in step S305.

[0157] It should be further explained that the cosine of the angle means whether the two vectors are roughly pointing in the same direction. The value range of the cosine of the angle is [-1,1]. The larger the cosine of the angle between the two vectors, the closer the directions of the two vectors are.

[0158] In the process of specifically implementing step S402, the cosine value of the first angle between the semantic vector of the voice instruction to be processed and the semantic vector of each sample interaction path is calculated to obtain multiple first cosine values ​​of the angle, that is, the cosine value of the first angle between the semantic vector of the voice instruction to be processed and the semantic vector of each sample interaction path is calculated.

[0159] The cosine value of the second angle between the semantic vector of the voice instruction to be processed and the semantic vector of each instruction template is calculated to obtain multiple cosine values ​​of the second angle, that is, the cosine value of the second angle between the semantic vector of the voice instruction to be processed and the semantic vector of each instruction template is calculated.

[0160] Step S403: Based on all the first angle cosine values ​​and the second angle cosine values, the standardized scores of the first angle cosine values ​​are calculated to obtain the similarity between the voice instruction to be processed and each sample interaction path, and the standardized scores of the second angle cosine values ​​are calculated to obtain the similarity between the voice instruction to be processed and each instruction template.

[0161] In the specific implementation of step S403, the average value and the standard deviation of all the first angle cosine values ​​and all the second angle cosine values ​​are calculated.

[0162] For each first angle cosine value, the first angle cosine value is first subtracted from the average value and then divided by the standard deviation to obtain a standardized score of the first angle cosine value. The standardized score of the first angle cosine value is used as the similarity between the voice command to be processed and the sample interaction path.

[0163] For each second angle cosine value, the average value is first subtracted from the second angle cosine value, and then divided by the standard deviation to obtain a standardized score for the second angle cosine value. The standardized score for the second angle cosine value is used as the similarity between the voice command to be processed and the command template.

[0164] By calculating the standardized score, the mean of all angle cosine values ​​(including the first angle cosine value and the second angle cosine value) can be converted to 0 and the variance thereof can be converted to 1.

[0165] Step S404: taking the sample interaction paths with similarity greater than 0 as candidate interaction paths, and taking the sample interaction paths corresponding to the instruction templates with similarity greater than 0 as candidate interaction paths.

[0166] In the process of specifically implementing step S404, after determining the similarity between the voice instruction to be processed and the sample interaction path, and determining the similarity between the voice instruction to be processed and the instruction template, the sample interaction path with a similarity greater than 0 is used as a candidate interaction path, and the sample interaction path corresponding to the instruction template with a similarity greater than 0 is used as a candidate interaction path.

[0167] In an embodiment of the present invention, the similarity between the voice instruction to be processed and the sample interaction path is obtained by calculating the standardized score of the cosine value of the angle between the semantic vector of the voice instruction to be processed and the semantic vector of the sample interaction path, and the similarity between the voice instruction to be processed and the instruction template is calculated in the same way. The sample interaction path with a similarity greater than 0 and the sample interaction path corresponding to the instruction template are taken as candidate interaction paths, and the interaction path to be processed is determined from the candidate interaction paths; the intelligent electronic device is operated according to the interaction path to be processed. It is realized that the intelligent electronic device is operated in a voice operation mode to trigger the corresponding function, which reduces the difficulty of operating the intelligent electronic device and improves the efficiency of operating the intelligent electronic device.

[0168] The above embodiments of the present invention Figure 1 For the process of constructing a decision tree involved in step S105, see Figure 5 , shows a flow chart of building a decision tree provided by an embodiment of the present invention, including the following steps:

[0169] Step S501: construct a bag-of-words model corresponding to each candidate interaction path.

[0170] In the specific implementation process of step S501, a bag-of-words model is constructed for each candidate interaction path. A bag-of-words model is constructed for each candidate interaction path.

[0171] It can be understood that the bag-of-words model of a candidate interaction path is used to convert the candidate interaction path into a vector, each dimension of the vector corresponds to a word, and a value of 1 for a dimension of the vector indicates that the candidate interaction path corresponding to the vector contains the word corresponding to the dimension with a value of 1, and a value of 0 for a dimension of the vector indicates that the candidate interaction path corresponding to the vector does not contain the word corresponding to the dimension with a value of 0. Therefore, the meaning of each dimension of the bag-of-words model is whether the corresponding text will appear in the process of using the graphical interface of an intelligent electronic device to complete the function corresponding to the candidate interaction path corresponding to the bag-of-words model.

[0172] To make it easier to understand the above contents about the dimensions of the bag-of-words model, Figure 6 For example, Figure 6 The schematic diagram of each dimension of the bag-of-words model is shown. Figure 6 It contains three candidate interaction paths, namely "WeChat-Address Book-Parameter-Message-Parameter-Send", "WeChat-Address Book-Parameter-Audio and Video Call-Voice Call" and "WeChat-Address Book-Parameter-Audio and Video Call-Video Call". The words corresponding to each dimension of the bag-of-words model of each candidate interaction path are: WeChat, Address Book, Message, Send, Audio and Video Call, Voice Call and Video Call.

[0173] For the candidate interaction path of "WeChat-Address Book-Parameter-Message-Parameter-Send", the values ​​of each dimension of the bag-of-words model of the candidate interaction path are: the value of the dimension corresponding to the word WeChat is 1 (represented by WeChat-1, and the same applies below), Address Book-1, Message-1, Send-1, Audio and Video Call-0, Voice Call-0, Video Call-0, indicating that the candidate interaction path includes the words WeChat, Address Book, Message, and Send.

[0174] For the candidate interaction path of "WeChat-Address Book-Parameters-Audio and Video Call-Voice Call", the values ​​of each dimension of the bag-of-words model of the candidate interaction path are: WeChat-1, Address Book-1, Send Message-0, Send-0, Audio and Video Call-1, Voice Call-1, Video Call-0, indicating that the candidate interaction path contains the words WeChat, Address Book, Audio and Video Call, and Voice Call.

[0175] For the candidate interaction path of "WeChat-Address Book-Parameters-Audio and Video Call-Video Call", the values ​​of each dimension of the bag-of-words model of the candidate interaction path are: WeChat-1, Address Book-1, Send Message-0, Send-0, Audio and Video Call-1, Voice Call-0, Video Call-1, indicating that the candidate interaction path contains the words WeChat, Address Book, Audio and Video Call, and Video Call.

[0176] Step S502: extract non-parameter texts from all candidate interaction paths and construct a text set.

[0177] It should be noted that each element in the text collection corresponds to a dimension in the vector space of the bag-of-words model.

[0178] In the specific implementation of step S502, non-parameter texts in all candidate interaction paths are extracted, and a text set is constructed based on each non-parameter text. The text set includes the non-parameter texts of all candidate interaction paths, such as Figure 6 In the example of the schematic diagram of each dimension of the bag-of-words model shown, the non-parametric texts contained in the text set are non-parametric texts such as WeChat, address book, message, send, audio and video call, voice call and video call.

[0179] Step S503: Take each candidate interaction path as a sample, take the similarity between each candidate interaction path and the voice instruction to be processed as a weight, and take the vector representation of the bag-of-words model of each candidate interaction path as a feature to construct a weighted decision tree.

[0180] It should be noted that each leaf node in the decision tree corresponds to a candidate interaction path, and each internal node in the decision tree corresponds to a non-parametric text.

[0181] In the specific implementation process of step S503, each candidate interaction path is taken as a sample, the similarity between each candidate interaction path and the voice instruction to be processed is taken as a weight, the vector representation of the bag-of-words model corresponding to each candidate interaction path is taken as a feature, and a weighted decision tree is constructed using the ID3 algorithm.

[0182] It should be noted that the ID3 algorithm is a decision tree construction algorithm. The ID3 algorithm selects the feature with the largest information entropy gain to segment the sample and generates a smaller tree structure based on the greedy algorithm. Similarly, other algorithms can also be used to construct a decision tree, which is not specifically limited in the embodiments of the present invention.

[0183] It can be understood that there are two types of nodes in the decision tree, namely internal nodes and leaf nodes. Each internal node in the decision tree corresponds to a dimension of the vector space. Since the vector dimension of the bag-of-words model corresponds one-to-one to the non-parametric text, each internal node of the decision tree corresponds to a non-parametric text. Each leaf node of the decision tree corresponds to a candidate interaction path. Each internal node of the decision tree points to the subsequent internal node or leaf node according to the different values ​​of the sample on the corresponding dimension.

[0184] In an embodiment of the present invention, a bag-of-words model is constructed for each candidate interaction path, and non-parametric texts in the candidate interaction path are extracted to construct a text set. A decision tree is constructed based on the candidate interaction path, the bag-of-words model of the candidate interaction path, and the text set. The interaction path to be processed is determined from the candidate interaction path in the decision tree, and the intelligent electronic device is operated according to the interaction path to be processed. The intelligent electronic device is operated in a voice-operated manner to trigger the corresponding function, thereby reducing the difficulty of operating the intelligent electronic device and improving the efficiency of operating the intelligent electronic device.

[0185] Corresponding to the device operation method based on voice provided in the above embodiment of the present invention, see Figure 7 , the embodiment of the present invention also provides a structural block diagram of a voice-based device operating system, the device operating system includes: a parsing unit 701, a matching unit 702, a processing unit 703 and an operating unit 704;

[0186] The parsing unit 701 is used to parse the voice data to be processed by the user for operating the intelligent electronic device to obtain the corresponding voice command to be processed.

[0187] The matching unit 702 is used to match the voice instruction to be processed using at least one command template when there is at least one preset command template, each command template has a corresponding predetermined sample interaction path, and each sample interaction path is operation-related information for triggering a corresponding function of the intelligent electronic device.

[0188] Preferably, the instruction template is generated by historical voice instructions of a first sample user for operating a corresponding intelligent electronic device, and the first sample user may or may not include the user to be processed.

[0189] Preferably, the sample interaction path is determined based on a pre-constructed interface jump relationship diagram, and the interface jump relationship diagram is constructed based on pre-collected operation-related information of the second sample user operation corresponding to the smart electronic device, and the second sample user includes or does not include the user to be processed.

[0190] The processing unit 703 is used to, if the target instruction template is not matched from at least one instruction template, or when at least one instruction template does not exist, execute the following steps to determine the interaction path to be processed, and execute the operation unit 704;

[0191] The following steps include:

[0192] Calculating the similarity between the voice command to be processed and each sample interaction path, and determining at least one candidate interaction path from each sample interaction path based on the similarity;

[0193] constructing a decision tree using at least one candidate interaction path;

[0194] Based on the decision tree, an inquiry message is played to the user to be processed, voice reply messages of the user to be processed are collected in real time, and the interaction path to be processed is determined from at least one candidate interaction path through the decision tree and the voice reply messages collected in real time.

[0195] In a specific implementation, the processing unit 704 for determining the interaction path to be processed is specifically used to: traverse from the root node of the decision tree, play an inquiry message to the user to be processed each time entering any internal node of the decision tree, and collect voice reply messages from the user to be processed in real time; determine the next node of the decision tree to be entered based on the voice reply messages collected in real time, repeat the step of playing the inquiry message to the user to be processed until entering any leaf node of the decision tree, and use the candidate interaction path corresponding to the leaf node entered as the interaction path to be processed.

[0196] Preferably, the processing unit 703 is also used to: play the interaction path information corresponding to the interaction path to be processed to the user to be processed, and collect the voice confirmation message of the user to be processed in real time; if the voice confirmation message indicates that the user to be processed denies the interaction path to be processed, delete the interaction path to be processed from all candidate interaction paths, and return to the step of building a decision tree using at least one candidate interaction path.

[0197] The operating unit 704 is used to operate the intelligent electronic device according to the interaction path to be processed.

[0198] In a specific implementation, the operation unit 704 is specifically used to: determine all operations and the order of operations in the interaction path to be processed; and execute the operations in the interaction path to be processed on the graphical interface of the intelligent electronic device according to the order of operations in the interaction path to be processed.

[0199] Preferably, the operating unit 704 is further used to: delete the function parameters in the voice instruction to be processed, and construct an instruction template based on the remaining part of the voice instruction to be processed.

[0200] In an embodiment of the present invention, the voice data of the user to be processed for operating the intelligent electronic device is converted into the voice command to be processed; when there is a preset command template, the command template is used to match the voice command to be processed; if the target command template cannot be matched, or the command template does not exist, the interaction path to be processed is determined from multiple sample interaction paths determined based on the pre-built interface jump relationship diagram; the intelligent electronic device is operated according to the interaction path to be processed. The intelligent electronic device is operated in a voice operation mode to trigger the corresponding function, which reduces the difficulty of operating the intelligent electronic device and improves the efficiency of operating the intelligent electronic device.

[0201] Preferably, combined Figure 7 The device operating system also includes:

[0202] The determination unit is configured to, if a target instruction template corresponding to the voice instruction to be processed is matched from at least one instruction template, use the sample interaction path corresponding to the target instruction template as the interaction path to be processed of the voice instruction to be processed, and execute the operation unit 704.

[0203] Preferably, combined Figure 7 The content shown, the processing unit 704 for determining the sample interaction path includes: a first construction module, a page clustering module, an element clustering module, a second construction module and a processing module, and the execution principle of each module is as follows:

[0204] The first construction module is used to construct an operation list based on the operation-related information of the smart electronic device corresponding to the second sample user operation collected in advance, and each operation-related information in the operation list at least includes: a page and a page layout corresponding to the operation-related information, and the page layout at least includes the elements in the corresponding page.

[0205] The page clustering module is used to cluster the pages in the operation list to obtain corresponding page clustering results.

[0206] The element clustering module is used to cluster the elements contained in each type of page in the operation list based on the page clustering results to obtain corresponding element clustering results.

[0207] The second construction module is used to construct an interface jump relationship graph using page clustering results and element clustering results. Each node in the interface jump relationship graph is an element in the operation list. Each node in the interface jump relationship graph has at least one edge pointing from other nodes to itself, and the node corresponding to each element in the operation list has at least one edge with itself as the starting node.

[0208] The processing module is used to segment the operation list based on the interface jump relationship diagram and the page clustering results, combined with the information indicating the repeated occurrence of pages in the operation list, to obtain multiple sample interaction paths.

[0209] Preferably, the processing module is also used to: for each sample interaction path, determine the input parameters corresponding to the sample interaction path, the source of the input parameters and the nature of the input parameters, and the nature of the input parameters is used to indicate that the input parameters are list selection parameters or text input parameters.

[0210] In an embodiment of the present invention, an operation list is constructed using the operation-related information of the second sample user collected in advance, and the operation list is converted into an interface jump relationship diagram. The operation list is segmented according to the interface jump relationship diagram to obtain multiple sample interaction paths. The multiple sample interaction paths are used to determine the interaction path to be processed corresponding to the voice instruction to be processed, and the intelligent electronic device is operated according to the interaction path to be processed. The intelligent electronic device is operated in a voice operation mode to trigger the corresponding function, thereby reducing the difficulty of operating the intelligent electronic device and improving the efficiency of operating the intelligent electronic device.

[0211] Preferably, combined Figure 7 The content shown, the processing unit 704 for determining the candidate interaction path includes: a determination module, a first calculation module, a second calculation module and a selection module, and the execution principle of each module is as follows:

[0212] The determination module is used to determine the semantic vector of the voice command to be processed by using a preset pre-trained model.

[0213] The first calculation module is used to calculate the cosine value of the first angle between the semantic vector of the voice instruction to be processed and the semantic vector of each predetermined sample interaction path, and to calculate the cosine value of the second angle between the semantic vector of the voice instruction to be processed and the semantic vector of each instruction template, the semantic vector of the sample interaction path is predetermined, and the semantic vector of the instruction template is predetermined.

[0214] In a specific implementation, the sample interaction path includes at least multiple elements and their operation order. The first calculation module for determining the semantic vector of the sample interaction path is specifically used to: extract the text information of each element in the sample interaction path and arrange them in the order of operation to obtain a keyword sequence; concatenate the keywords in the keyword sequence into a concatenated sentence; use a preset pre-trained model to calculate the word vector of each keyword in the keyword sequence, add all the word vectors to obtain the word vector part, and embed the concatenated sentence to obtain the sentence vector part; normalize the word vector part and the sentence vector part and add them in equal proportion to obtain the semantic vector of the sample interaction path.

[0215] The second calculation module is used to calculate the standardized scores of each first angle cosine value based on all the first angle cosine values ​​and the second angle cosine values ​​to obtain the similarity between the voice instruction to be processed and each sample interaction path, and to calculate the standardized scores of each second angle cosine value to obtain the similarity between the voice instruction to be processed and each instruction template.

[0216] The selection module is used to take the sample interaction paths with a similarity greater than 0 as candidate interaction paths, and take the sample interaction paths corresponding to the instruction templates with a similarity greater than 0 as candidate interaction paths.

[0217] In an embodiment of the present invention, the similarity between the voice instruction to be processed and the sample interaction path is obtained by calculating the standardized score of the cosine value of the angle between the semantic vector of the voice instruction to be processed and the semantic vector of the sample interaction path, and the similarity between the voice instruction to be processed and the instruction template is calculated in the same way. The sample interaction path with a similarity greater than 0 and the sample interaction path corresponding to the instruction template are taken as candidate interaction paths, and the interaction path to be processed is determined from the candidate interaction paths; the intelligent electronic device is operated according to the interaction path to be processed. It is realized that the intelligent electronic device is operated in a voice operation mode to trigger the corresponding function, which reduces the difficulty of operating the intelligent electronic device and improves the efficiency of operating the intelligent electronic device.

[0218] Preferably, combined Figure 7 The processing unit 704 for constructing a decision tree includes: a third construction module, an extraction module and a fourth construction module. The execution principle of each module is as follows:

[0219] The third building module is used to build a bag-of-words model corresponding to each candidate interaction path.

[0220] The extraction module is used to extract non-parametric texts from all candidate interaction paths and construct a text set, where each element in the text set corresponds to a dimension in the vector space of the bag-of-words model.

[0221] The fourth construction module is used to take each candidate interaction path as a sample, the similarity between each candidate interaction path and the voice instruction to be processed as a weight, and the vector representation of the bag-of-words model of each candidate interaction path as a feature to construct a weighted decision tree, where each leaf node in the decision tree corresponds to a candidate interaction path, and each internal node in the decision tree corresponds to a non-parametric text.

[0222] In an embodiment of the present invention, a bag-of-words model is constructed for each candidate interaction path, and non-parametric texts in the candidate interaction path are extracted to construct a text set. A decision tree is constructed based on the candidate interaction path, the bag-of-words model of the candidate interaction path, and the text set. The interaction path to be processed is determined from the candidate interaction path in the decision tree, and the intelligent electronic device is operated according to the interaction path to be processed. The intelligent electronic device is operated in a voice-operated manner to trigger the corresponding function, thereby reducing the difficulty of operating the intelligent electronic device and improving the efficiency of operating the intelligent electronic device.

[0223] In summary, the embodiments of the present invention provide a device operation method and system based on voice, which converts the voice data of the user to be processed for operating the smart electronic device into the voice instructions to be processed. When there is a preset instruction template, the instruction template is used to match the voice instructions to be processed. If the target instruction template cannot be matched, or the instruction template does not exist, the interaction path to be processed is determined from multiple sample interaction paths determined based on a pre-built interface jump relationship diagram. The smart electronic device is operated according to the interaction path to be processed. It is realized that the smart electronic device is operated in a voice operation mode to trigger the corresponding function, which reduces the difficulty of operating the smart electronic device and improves the efficiency of operating the smart electronic device.

[0224] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0225] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0226] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A device operation method based on voice, characterized in that: The method comprises: Analyze the voice data of the user to be processed for operating the intelligent electronic device to obtain the corresponding voice command to be processed; When there is at least one preset instruction template, the voice instruction to be processed is matched using the at least one instruction template, each instruction template has a corresponding predetermined sample interaction path, and each sample interaction path is operation-related information for triggering a corresponding function of the intelligent electronic device; If the target instruction template is not matched from the at least one instruction template, or when the at least one instruction template does not exist, the following steps are performed to determine the interaction path to be processed; operating the intelligent electronic device according to the interaction path to be processed; The following steps include: Calculating the similarity between the voice instruction to be processed and each of the sample interaction paths, and determining at least one candidate interaction path from each of the sample interaction paths based on the similarity; constructing a decision tree using the at least one candidate interaction path; Based on the decision tree, an inquiry message is played to the user to be processed, voice reply messages of the user to be processed are collected in real time, and the interaction path to be processed is determined from the at least one candidate interaction path through the decision tree and the voice reply messages collected in real time.

2. The method according to claim 1, characterized in that The method further comprises: If a target instruction template corresponding to the voice instruction to be processed is obtained by matching from the at least one instruction template, the sample interaction path corresponding to the target instruction template is used as the interaction path to be processed of the voice instruction to be processed.

3. The method according to claim 1, characterized in that The instruction template is generated by historical voice instructions of a first sample user for operating a corresponding intelligent electronic device, and the first sample user includes or does not include the user to be processed.

4. The method according to claim 1, characterized in that: The sample interaction path is determined based on a pre-constructed interface jump relationship diagram, and the interface jump relationship diagram is constructed based on pre-collected operation-related information of the smart electronic device corresponding to the second sample user operation, and the second sample user includes or does not include the user to be processed.

5. The method according to claim 4, characterized in that The process of determining sample interaction paths includes: Based on the pre-collected operation-related information of the intelligent electronic device corresponding to the second sample user operation, an operation list is constructed, wherein each operation-related information in the operation list at least includes: a page and a page layout corresponding to the operation-related information, wherein the page layout at least includes elements in the corresponding page; Clustering the pages in the operation list to obtain corresponding page clustering results; Based on the page clustering result, clustering the elements contained in each type of page in the operation list to obtain corresponding element clustering results; Using the page clustering results and the element clustering results, an interface jump relationship graph is constructed, wherein each node in the interface jump relationship graph is an element in the operation list, each node in the interface jump relationship graph has at least one edge pointing from other nodes to itself, and each node corresponding to each element in the operation list has at least one edge with itself as the starting node; Based on the interface jump relationship diagram and the page clustering result, combined with the information indicating the repeated occurrence of pages in the operation list, the operation list is segmented to obtain a plurality of sample interaction paths.

6. The method according to claim 5, characterized in that After obtaining the plurality of sample interaction paths, the method further includes: For each sample interaction path, the input parameter corresponding to the sample interaction path, the source of the input parameter and the property of the input parameter are determined, and the property of the input parameter is used to indicate that the input parameter is a list selection parameter or a text input parameter.

7. The method according to claim 1, characterized in that Calculating the similarity between the voice instruction to be processed and each of the sample interaction paths, and determining at least one candidate interaction path from each of the sample interaction paths based on the similarity, including: Determining the semantic vector of the voice command to be processed by using a preset pre-trained model; Calculating a cosine value of a first angle between a semantic vector of the voice instruction to be processed and a semantic vector of each predetermined sample interaction path, and calculating a cosine value of a second angle between the semantic vector of the voice instruction to be processed and a semantic vector of each instruction template, wherein the semantic vector of the sample interaction path is predetermined and the semantic vector of the instruction template is predetermined; Based on all the first angle cosine values ​​and the second angle cosine values, a standardized score of each of the first angle cosine values ​​is calculated to obtain the similarity between the voice instruction to be processed and each of the sample interaction paths, and a standardized score of each of the second angle cosine values ​​is calculated to obtain the similarity between the voice instruction to be processed and each of the instruction templates; The sample interaction paths with the similarity greater than 0 are taken as candidate interaction paths, and the sample interaction paths corresponding to the instruction templates with the similarity greater than 0 are taken as candidate interaction paths.

8. The method according to claim 7, characterized in that The sample interaction path at least includes a plurality of elements and their operation sequence, and the process of determining the semantic vector of the sample interaction path includes: Extracting text information of each element in the sample interaction path and arranging them in operation order to obtain a keyword sequence; splicing each keyword in the keyword sequence into a spliced ​​sentence; Using a preset pre-trained model, the word vector of each keyword in the keyword sequence is calculated, all the word vectors are added together to obtain a word vector part, and the concatenated sentence is embedded to obtain a sentence vector part; The word vector part and the sentence vector part are unitized and added in equal proportion to obtain the semantic vector of the sample interaction path.

9. The method according to claim 1, characterized in that: The step of constructing a decision tree using the at least one candidate interaction path includes: Construct a bag-of-words model for each candidate interaction path; Extracting non-parametric texts from all the candidate interaction paths and constructing a text set, each element in the text set corresponds to a dimension in the vector space of the bag-of-words model; Taking each of the candidate interaction paths as a sample, taking the similarity between each of the candidate interaction paths and the voice instruction to be processed as a weight, and taking the vector representation of the bag-of-words model of each of the candidate interaction paths as a feature, a weighted decision tree is constructed, wherein each leaf node in the decision tree corresponds to one of the candidate interaction paths, and each internal node in the decision tree corresponds to one of the non-parametric texts.

10. The method according to claim 9, characterized in that The step of playing the inquiry message to the user to be processed based on the decision tree, collecting voice reply messages of the user to be processed in real time, and determining the interaction path to be processed from the at least one candidate interaction path through the decision tree and the voice reply messages collected in real time, comprises: Traversing from the root node of the decision tree, playing a query message to the user to be processed each time entering any internal node of the decision tree, and collecting voice reply messages of the user to be processed in real time; Based on the voice reply message collected in real time, the next node of the decision tree to be entered is determined, and the step of playing the inquiry message to the user to be processed is repeated until any leaf node of the decision tree is entered, and the candidate interaction path corresponding to the leaf node entered is used as the interaction path to be processed.

11. The method according to claim 1, characterized in that: After determining the to-be-processed interaction path from the at least one candidate interaction path, the method further includes: Playing the interaction path information corresponding to the interaction path to be processed to the user to be processed, and collecting the voice confirmation message of the user to be processed in real time; If the voice confirmation message indicates that the to-be-processed user denies the to-be-processed interaction path, the to-be-processed interaction path is deleted from all the candidate interaction paths, and the process returns to the step of constructing a decision tree using the at least one candidate interaction path.

12. The method according to claim 1, characterized in that The operating the intelligent electronic device according to the interaction path to be processed includes: Determine all operations and their order in the interaction path to be processed; According to the operation sequence in the interaction path to be processed, the operations in the interaction path to be processed are executed on the graphic interface of the intelligent electronic device.

13. The method according to claim 1, characterized in that After operating the intelligent electronic device according to the interaction path to be processed, the method further includes: The function parameters in the voice instruction to be processed are deleted, and a command template is constructed based on the remaining part of the voice instruction to be processed.

14. A voice-based device operating system, characterized in that: The system comprises: The parsing unit is used to parse the voice data of the user to be processed for operating the intelligent electronic device to obtain the corresponding voice command to be processed; A matching unit, configured to match the voice command to be processed by using at least one preset command template when there is at least one preset command template, each of the command templates having a corresponding predetermined sample interaction path, each of the sample interaction paths being operation-related information for triggering a corresponding function of the intelligent electronic device; A processing unit, configured to, if no target instruction template is matched from the at least one instruction template, or when the at least one instruction template does not exist, execute the following steps to determine an interaction path to be processed, and execute an operation unit; The following steps include: Calculating the similarity between the voice instruction to be processed and each of the sample interaction paths, and determining at least one candidate interaction path from each of the sample interaction paths based on the similarity; constructing a decision tree using the at least one candidate interaction path; Playing an inquiry message to the user to be processed based on the decision tree, collecting voice reply messages of the user to be processed in real time, and determining the interaction path to be processed from the at least one candidate interaction path through the decision tree and the voice reply messages collected in real time; The operating unit is used to operate the intelligent electronic device according to the interaction path to be processed.

Images