Instruction execution method, device, storage medium and electronic device

Abstract

The embodiment of the present application discloses an instruction execution method, device, storage medium and electronic device, wherein, when receiving a user instruction, the embodiment of the present application generates a first feature vector according to the user instruction; obtains current panoramic data, and generates a second feature vector according to the panoramic data; fuses the first feature vector and the second feature vector to generate a user feature matrix; obtains an intent label matching the user instruction according to the user feature matrix and a pre-trained intent prediction model; and executes the user instruction according to the intent label. The solution supplements the context information and situation information of the user instruction by collecting panoramic data, and can more accurately understand the user's implicit intent, thereby better executing the user instruction.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an instruction execution method, device, storage medium and electronic device. Background Art

[0002] Users generally control the terminal to perform corresponding operations through control commands, such as voice commands, touch operation commands, etc. With the development of terminal technology, the terminal needs to understand the implicit intentions of the control commands issued by the user, and then better execute the user commands. The existing intention understanding solutions mainly use speech recognition technology, combined with mixed Gaussian models, temporal classification models, etc. to analyze speech data and extract key words from the commands as the understanding of implicit intentions; however, this method of user intention recognition is only based on the user's voice commands, which makes it difficult to accurately understand and describe the user's actual needs. Summary of the invention

[0003] The embodiments of the present application provide a method, device, storage medium and electronic device for executing an instruction, which can improve the accuracy of a terminal in identifying a user's implicit intention, so as to more accurately execute user instructions.

[0004] In a first aspect, an embodiment of the present application provides an instruction execution method, comprising:

[0005] When receiving a user instruction, generating a first feature vector according to the user instruction;

[0006] Acquire current panoramic data, and generate a second feature vector according to the panoramic data;

[0007] fusing the first feature vector and the second feature vector to generate a user feature matrix;

[0008] Acquire an intent label matching the user instruction according to the user feature matrix and a pre-trained intent prediction model;

[0009] The user instruction is executed according to the intention tag.

[0010] In a second aspect, an embodiment of the present application provides an instruction execution device, comprising:

[0011] A signal feature extraction module, configured to generate a first feature vector according to a user instruction when receiving the user instruction;

[0012] A scene feature extraction module, used to obtain current panoramic data and generate a second feature vector according to the panoramic data;

[0013] A feature fusion module, used for fusing the first feature vector and the second feature vector to generate a user feature matrix;

[0014] An intention prediction module, used to obtain an intention label matching the user instruction according to the user feature matrix and a pre-trained intention prediction model;

[0015] An instruction execution module is used to execute the user instruction according to the intention tag.

[0016] In a third aspect, the storage medium provided in the embodiments of the present application stores a computer program thereon, and when the computer program runs on a computer, the computer executes the instruction execution method provided in any embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory has a computer program, and the processor executes an instruction execution method provided in any embodiment of the present application by calling the computer program.

[0018] The technical solution provided by the embodiment of the present application, when receiving a user instruction, generates a first feature vector according to the user instruction, obtains current panoramic data, generates a second feature vector according to the panoramic data, fuses the first feature vector and the second feature vector to generate a user feature matrix, then uses the user feature matrix as input data of a pre-trained intention prediction model, obtains an intention label matching the user instruction, and executes the user instruction according to the intention label. This solution supplements the context information and situation information of the user instruction by collecting panoramic data, so as to more accurately understand the user's implicit intention and thus better execute the user instruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of a panoramic perception architecture of the instruction execution method provided in an embodiment of the present application.

[0021] Figure 2 A schematic diagram of an application scenario of the instruction execution method provided in an embodiment of the present application.

[0022] Figure 3 A first flow chart of the instruction execution method provided in an embodiment of the present application.

[0023] Figure 4 A second flow chart of the instruction execution method provided in an embodiment of the present application.

[0024] Figure 5 A third flow chart of the instruction execution method provided in an embodiment of the present application.

[0025] Figure 6 A schematic diagram of the structure of an instruction execution device provided in an embodiment of the present application.

[0026] Figure 7 A first structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0027] Figure 8 A second structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0029] The terms "first", "second", and "third" in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules that are not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.

[0030] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] refer to Figure 1 , Figure 1 A schematic diagram of a panoramic perception architecture of an instruction execution method provided in an embodiment of the present application. The instruction execution method is applied to an electronic device. The electronic device is provided with a panoramic perception architecture. The panoramic perception architecture is an integration of hardware and software used to implement the instruction execution method in the electronic device.

[0032] Among them, the panoramic perception architecture includes information perception layer, data processing layer, feature extraction layer, scenario modeling layer and intelligent service layer.

[0033] The information perception layer is used to obtain information about the electronic device itself or information in the external environment. The information perception layer may include multiple sensors. For example, the information perception layer includes multiple sensors such as distance sensors, magnetic field sensors, light sensors, acceleration sensors, fingerprint sensors, Hall sensors, position sensors, gyroscopes, inertial sensors, attitude sensors, barometers, heart rate sensors, etc.

[0034] Among them, the distance sensor can be used to detect the distance between the electronic device and the external object. The magnetic field sensor can be used to detect the magnetic field information of the environment in which the electronic device is located. The light sensor can be used to detect the light information of the environment in which the electronic device is located. The acceleration sensor can be used to detect the acceleration data of the electronic device. The fingerprint sensor can be used to collect the fingerprint information of the user. The Hall sensor is a magnetic field sensor made according to the Hall effect, which can be used to realize the automatic control of the electronic device. The position sensor can be used to detect the current geographical location of the electronic device. The gyroscope can be used to detect the angular velocity of the electronic device in all directions. The inertial sensor can be used to detect the motion data of the electronic device. The attitude sensor can be used to sense the attitude information of the electronic device. The barometer can be used to detect the air pressure of the environment in which the electronic device is located. The heart rate sensor can be used to detect the heart rate information of the user.

[0035] The data processing layer is used to process the data obtained by the information perception layer. For example, the data processing layer can perform data cleaning, data integration, data transformation, data reduction, etc. on the data obtained by the information perception layer.

[0036] Among them, data cleaning refers to cleaning the large amount of data obtained by the information perception layer to eliminate invalid data and duplicate data. Data integration refers to integrating multiple single-dimensional data obtained by the information perception layer into a higher or more abstract dimension to comprehensively process multiple single-dimensional data. Data transformation refers to converting the data type or format of the data obtained by the information perception layer so that the transformed data meets the processing requirements. Data reduction refers to maximizing the reduction of data volume while keeping the original appearance of the data as much as possible.

[0037] The feature extraction layer is used to extract features from the data processed by the data processing layer to extract features included in the data. The extracted features can reflect the state of the electronic device itself, the state of the user, or the environmental state of the environment in which the electronic device is located.

[0038] Among them, the feature extraction layer can extract features or process the extracted features through filtering, packaging, integration and other methods.

[0039] The filtering method refers to filtering the extracted features to delete redundant feature data. The packaging method is used to screen the extracted features. The integration method refers to integrating multiple feature extraction methods together to build a more efficient and accurate feature extraction method for extracting features.

[0040] The scenario modeling layer is used to construct a model based on the features extracted by the feature extraction layer, and the obtained model can be used to represent the state of the electronic device, the state of the user, or the state of the environment, etc. For example, the scenario modeling layer can construct a key value model, a pattern identification model, a graph model, an entity relationship model, an object-oriented model, etc. based on the features extracted by the feature extraction layer.

[0041] The intelligent service layer is used to provide intelligent services to users based on the model constructed by the scenario modeling layer. For example, the intelligent service layer can provide basic application services to users, perform system intelligent optimization for electronic devices, and provide personalized intelligent services to users.

[0042] In addition, the panoramic perception architecture may also include multiple algorithms, each of which can be used to analyze and process data, and the multiple algorithms may constitute an algorithm library. For example, the algorithm library may include Markov algorithm, invisible Dirichlet distribution algorithm, Bayesian classification algorithm, support vector machine, K-means clustering algorithm, K-nearest neighbor algorithm, conditional random field, residual network, long short-term memory network, convolutional neural network, recurrent neural network and other algorithms.

[0043] Based on the above panoramic perception architecture, the electronic device collects the user's panoramic data through the information perception layer and / or other means, and the data processing layer processes the panoramic data, for example, data cleaning and data integration of the acquired panoramic data. Next, the intelligent service layer responds to the user's instruction according to the instruction execution method proposed in this application. For example, when receiving the user's instruction, a first feature vector is generated according to the user's instruction, the current panoramic data is obtained, a second feature vector is generated according to the panoramic data, the first feature vector and the second feature vector are fused to generate a user feature matrix, and then the user feature matrix is ​​used as the input data of the pre-trained intent prediction model to obtain the intent label that matches the user's instruction, and the user instruction is executed according to the intent label. This solution supplements the context information and situation information of the user's instruction by collecting panoramic data, and can more accurately understand the user's implicit intention, thereby better executing the user's instruction.

[0044] The embodiment of the present application provides an instruction execution method, and the execution subject of the instruction execution method can be the instruction execution device provided in the embodiment of the present application, or an electronic device integrated with the instruction execution device, wherein the instruction execution device can be implemented in hardware or software. Among them, the electronic device can be a smart phone, a tablet computer, a PDA, a laptop computer, or a desktop computer.

[0045] See also Figure 2 , Figure 2 A schematic diagram of an application scenario of the instruction execution method provided in an embodiment of the present application, taking the example of an instruction execution device integrated in an electronic device, the electronic device can receive user instructions, such as voice instructions, touch instructions, or holding instructions, etc. When receiving the user instruction, a first feature vector is generated according to the user instruction; then the current panoramic data of the electronic device is collected, such as user information, sensor status data, usage information of application programs in the electronic device, etc., and a second feature vector is generated according to the collected panoramic data. Next, the first feature vector and the second feature vector are fused to generate a user feature matrix, and an intent label corresponding to the user feature matrix is ​​generated according to the user feature matrix and a pre-trained intent prediction model, and the user instruction is executed according to the intent label. In this way, according to the panoramic data collected when receiving the instruction, the user instruction is supplemented with contextual information and information of the scenario environment, so that the electronic device can more accurately understand the user's implicit intention, thereby better executing the user instruction.

[0046] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a first flow chart of the instruction execution method provided in the embodiment of the present application. The specific flow chart of the instruction execution method provided in the embodiment of the present application may be as follows:

[0047] Step 101: When a user instruction is received, a first feature vector is generated according to the user instruction.

[0048] In this embodiment, the user command can be of various types, such as voice commands, touch commands, and holding commands. When the electronic device receives the user command, it obtains the signal data corresponding to the user command, normalizes the signal data, generates the corresponding first feature vector, and uses the first feature vector to represent the information contained in the user command. The first feature vector can be expressed as follows:

[0049] s1={yi1,yi2,…,yi n}

[0050] For example, in one embodiment, a voice component, such as a microphone, is provided in the electronic device, and the electronic device can continuously collect the user's voice data through the microphone. After the voice recognition function of the electronic device is turned on, the user can control the electronic device through voice commands, such as "play music", "make a call" and other commands. For example, when a user command is received, the step of generating a first feature vector according to the user command includes: when a voice command is received, obtaining the voice data collected by the voice component; generating a semantic feature vector of the voice data according to a pre-trained autoencoder recurrent neural network, and using the semantic feature vector as the first feature vector.

[0051] Among them, the autoencoding neural network model consists of an encoder and a decoder. The output of the network is equal to the input. The network also includes an intermediate hidden layer, which can extract the semantic feature vector of the speech data. In this scheme, an autoencoding recurrent neural network is used to extract semantic feature vectors from speech data. The input data and output data of the autoencoding recurrent neural network are both the above-mentioned speech data. When training the network, there is no need to label the speech data. A large amount of speech data is collected in advance as the input and output of the network. The network determines the parameters through self-learning.

[0052] Alternatively, in another embodiment, other feature extraction methods may be used to obtain the speech feature vector. Step 101: when a user instruction is received, the step of generating a first feature vector according to the user instruction may include:

[0053] When receiving a voice command, obtain the voice data collected by the voice component;

[0054] Converting the speech data into a spectrogram according to an audio feature extraction algorithm;

[0055] A semantic feature vector of the speech data is generated according to a pre-trained autoencoder convolutional neural network and the spectrogram, and the semantic feature vector is used as the first feature vector.

[0056] Among them, the audio feature extraction algorithm can be an MFCC (Mel Frequency Cepstrum Coefficient) algorithm or an FFT (Fast Fourier Transformation) algorithm, and the speech data is converted into a spectrogram through the audio feature extraction algorithm, and the spectrogram is used as the input data and output data of the autoencoder convolutional neural network to extract the semantic feature vector from the network. Similar to the above-mentioned autoencoder recurrent neural network, the autoencoder convolutional neural network is also an autoencoder, wherein the autoencoder convolutional neural network uses a convolutional layer to construct an autoencoder, and the output data of the autoencoder is trained to be consistent with the input data to obtain valuable information in the middle hidden layer.

[0057] The electronic device obtains the semantic vector feature through the above solution and uses the semantic vector feature as the first feature vector.

[0058] For another example, in one embodiment, the user instruction is a touch instruction; when the user instruction is received, the step of generating a first feature vector according to the user instruction may include: when the touch instruction is received, acquiring touch data collected by the touch sensor; and converting the touch data into the first feature vector.

[0059] Among them, the electronic device is provided with a touch sensor for detecting touch operations, such as a touch sensor provided under a touch screen, which can detect various touch operations and touch gestures input by the user. When a touch command is received, the touch data collected by the touch sensor is obtained, and the touch data includes the touch position and the touch track.

[0060] For another example, in one embodiment, the user instruction is a holding instruction; when the user instruction is received, the step of generating a first feature vector according to the user instruction may include: when the holding instruction is received, obtaining holding data collected by the holding sensor; and converting the holding data into the first feature vector.

[0061] Among them, the frame, back cover and other positions of the electronic device are provided with a grip sensor, which can detect the grip instruction triggered by the user's grip operation. The grip data can be position information, such as position coordinates. In order to facilitate the subsequent fusion of multiple feature vectors into a feature matrix, it is necessary to predefine the preset length of the feature vector. The electronic device converts the acquired position coordinates into a vector representation of a preset length. For example, if the preset vector length is 10, the position coordinates with a length of 2 are converted into a first feature vector with a length of 10 by repeated superposition.

[0062] Step 102: Acquire current panoramic data, and generate a second feature vector according to the panoramic data.

[0063] In the embodiment of the present application, the panoramic data of the electronic device includes but is not limited to the following types of data: terminal status data, user status data and sensor status data.

[0064] The user status data includes the user's facial image indirectly captured by the camera of the electronic device, and the user's age, gender and other user information obtained from the user database.

[0065] The terminal operation data includes the operation mode of the user terminal in each time interval, where the operation mode includes game mode, entertainment mode, audio and video mode, etc. The operation mode of the terminal can be determined according to the type of the currently running application. The type of the currently running application can be directly obtained from the classification information of the application installation package; or, the terminal operation data can also include the remaining power of the terminal, display mode, network status, screen off / locked status, etc.

[0066] The sensor status data includes signals collected by various sensors on the electronic device. For example, the electronic device includes the following sensors: distance sensor, magnetic field sensor, light sensor, acceleration sensor, fingerprint sensor, Hall sensor, position sensor, gyroscope, inertial sensor, attitude sensor, barometer, heart rate sensor and other sensors. The sensor status data of the electronic device when receiving the user command is obtained, or the sensor status data of the electronic device for a period of time before receiving the user command is obtained. In some embodiments, the status data of some sensors can be obtained in a targeted manner.

[0067] See also Figure 4 , Figure 4 The second flow chart of the instruction execution method provided in the embodiment of the present application is as follows: Step 102, obtaining current panoramic data, and generating a second feature vector according to the panoramic data may include:

[0068] Step 1021, obtaining current terminal status data, user status data and sensor status data;

[0069] Step 1022: generating a terminal state feature according to the terminal state data, generating a user state feature according to the user state data, and generating a terminal scenario feature according to the sensor state data;

[0070] Step 1023: Fusion the terminal state feature, the user state feature, and the terminal scenario feature to generate the second feature vector.

[0071] Among them, in some embodiments, the step of obtaining user status data may include: calling a camera component to capture a user facial image; recognizing the user facial image according to a preset convolutional neural network model to generate a user emotion tag; obtaining user information, and using the user emotion tag and the user information as the user status data. The user information can be obtained from a user database, and the user information may include user gender, user age, user hobbies, etc.

[0072] Among them, in some embodiments, the step of obtaining the current terminal status data includes: determining the program category described by the currently running application; determining the current operating mode of the terminal according to the program category, wherein the operating mode includes a game mode, an entertainment mode, and an audio-visual mode; and using the operating mode as the terminal status data.

[0073] In summary, the terminal state feature ys1 is generated according to the terminal state data. According to the state data of the sensor, for example, the four-dimensional terminal posture features ys2 to ys5 are obtained from the magnetometer, accelerometer, and gyroscope through the Kalman filter algorithm, the air pressure feature ys6 is obtained through the data collected by the barometer, the WIFI connection status ys7 is determined through the network module, and the data collected by the position sensor is used for positioning to obtain the user's current location attributes (such as shopping malls, homes, companies, parks, etc.) and generate feature ys8; it can also be further combined with the magnetometer, accelerometer, gyroscope, barometer 10-axis information using a filtering algorithm or a principal component analysis algorithm to obtain new multidimensional data and generate the corresponding feature ys9. Generate feature ys according to the user's emotional label 10 , generate feature ys based on the user's gender, age, and hobbies 11 ~ys 13 . For the non-digital features in the above features, you can use the method of establishing index numbers to convert them into digital representations. For example, for the feature of the operating mode of the current system terminal state, use the index number to represent the current state mode, such as 1 for game mode, 2 for entertainment mode, and 3 for audio and video mode. If the current operating mode is game mode, then determine that the current system state ys1 = 1. After obtaining all the digitally represented features, merge the above feature data to obtain a long vector, normalize the long vector, and obtain the second feature vector s2:

[0074] s2={ys1,ys2,…,ys m}

[0075] Step 103: Fusing the first feature vector and the second feature vector to generate a user feature matrix.

[0076] The first eigenvector s1 and the second eigenvector s2 may be superimposed on each other to generate the following user feature matrix:

[0077]

[0078] If the lengths of the first eigenvector and the second eigenvector are not equal, the shorter vector can be adjusted by zero padding. If n < m, the length of the first eigenvector s1 is extended to m by zero padding. If n > m, the length of the second eigenvector s2 is extended to n by zero padding.

[0079] Alternatively, in an optional implementation, after adjusting the first eigenvector and the second eigenvector to the same length, the first eigenvector s1 and the second eigenvector s2 are matrix-superimposed, and then, in order to generate a richer eigenvector for subsequent operations, the superimposed matrix is ​​flipped to generate the following matrix:

[0080]

[0081] The matrices before and after the flipping operation are combined to obtain the following matrix as the user feature matrix.

[0082]

[0083] Step 104: Obtain an intent label that matches the user instruction based on the user feature matrix and a pre-trained intent prediction model.

[0084] In the embodiment of the present application, the intention prediction model is a classification model that characterizes the relationship between the user feature matrix and the intention label. For example, the intention prediction model can be obtained by training a classification algorithm such as a convolutional neural network, a BP neural network (Back Propagation) or an SVM (Support Vector Machine) algorithm. Taking the convolutional neural network as an example, a large amount of sample data of test users is collected, and features are extracted according to steps 101 to 103, and then the extracted features are labeled. For example, a part of users are first selected from all users as test users, and the user instructions of these users are recorded, and a first feature vector is generated; and the panoramic data when the user instruction is received is collected, and a second feature vector is generated according to the panoramic data. Then the response of the electronic device to the user instruction and the operation performed by the user based on the response are recorded. In an optional embodiment, the intention label data can be automatically generated according to the response of the electronic device to the user instruction and the operation performed by the user based on the response, or, in some other embodiments, the user feature matrix can be labeled by manual labeling.

[0085] The user feature matrix with intent labels is input into a convolutional neural network for training, wherein the structure and hyperparameters of the convolutional neural network can be pre-set by the user as needed. After training, the weight parameters of the network are determined to generate an intent prediction model.

[0086] In addition, it should be noted that in order to fully characterize user intentions, the intent label corresponding to a user feature matrix is ​​an intent label set, which contains multiple labels that can characterize user instructions, location, time, environment, status, habits and other information. These labels depict the user's current panoramic portrait.

[0087] By inputting the user feature matrix obtained in step 103 into the intention prediction model, an intention label matching the user instruction can be generated.

[0088] Step 105: Execute the user instruction according to the intention tag.

[0089] The intent label output by the intent prediction model is a label set composed of multiple labels. These labels constitute a panoramic portrait corresponding to the user's instructions. When the electronic device responds to the user's instructions and starts the corresponding target application, it pushes the label set to the target application. In this way, when the electronic device responds to the user's instructions, it performs more than just a simple operation such as starting the software. After the target application is started, it can further perform more specific operations based on the received label set.

[0090] For example, refer to Figure 5 As shown, Figure 5 The third flow chart of the instruction execution method provided in the embodiment of the present application is as follows: Step 105, executing the user instruction according to the intention tag includes:

[0091] Step 1051: Determine the target application corresponding to the user instruction according to the intention tag;

[0092] Step 1052: Start the target application and send the intent tag to the target application, wherein the target application performs a corresponding operation based on the intent tag.

[0093] For example, when a user triggers a voice command "Xiaoou Xiaoou, order me food", the system's voice recognition module detects the command and obtains more data that can reflect the user's intention through panoramic perception technology, such as terminal status data, user status data, and sensor status data, etc. These data are used to supplement the context information of the command to obtain more comprehensive intention information, such as: current time, current location, scene, user eating habits, user hunger, etc. When the electronic device opens a third-party ordering application according to the command, this information is pushed to the third-party ordering application, and the third-party ordering application can recommend specific ordering locations and ordering contents to the user based on the above information.

[0094] From the above, it can be seen that the instruction execution method proposed in the embodiment of the present application, when receiving a user instruction, generates a first feature vector according to the user instruction, obtains the current panoramic data, generates a second feature vector according to the panoramic data, fuses the first feature vector and the second feature vector, and generates a user feature matrix. Next, the user feature matrix is ​​used as input data of a pre-trained intention prediction model to obtain an intention label matching the user instruction, and executes the user instruction according to the intention label. This scheme supplements the context information and situational information of the user instruction by collecting panoramic data, so as to more accurately understand the user's implicit intention and thus better execute the user instruction.

[0095] In one embodiment, a device for executing instructions is also provided. Figure 6 , Figure 6 The schematic diagram of the structure of the instruction execution device 400 provided in the embodiment of the present application. The instruction execution device 400 is applied to an electronic device, and the instruction execution device 400 includes a first feature extraction module 401, a second feature extraction module 402, a feature fusion module 403, an intention prediction module 404 and an instruction execution module 405, as follows:

[0096] The first feature extraction module 401 is used to generate a first feature vector according to the user instruction when receiving the user instruction.

[0097] In this embodiment, the user command can be of various types, such as voice commands, touch commands, and holding commands. When receiving the user command, the first feature extraction module 401 obtains the signal data corresponding to the user command, normalizes the signal data, generates the corresponding first feature vector, and uses the first feature vector to characterize the information contained in the user command. The first feature vector can be expressed as follows:

[0098] s1={yi1,yi2,…,yi n}

[0099] For example, in one embodiment, a voice component, such as a microphone, is provided in the electronic device, and the electronic device can continuously collect the user's voice data through the microphone. After the voice recognition function of the electronic device is turned on, the user can control the electronic device through voice commands, such as "play music", "make a call" and other commands. For example, the first feature extraction module 401 is also used to: when receiving a voice command, obtain the voice data collected by the voice component; generate a semantic feature vector of the voice data according to a pre-trained autoencoder recurrent neural network, and use the semantic feature vector as the first feature vector.

[0100] Among them, the autoencoding neural network model consists of an encoder and a decoder. The output of the network is equal to the input. The network also includes an intermediate hidden layer, which can extract the semantic feature vector of the speech data. In this scheme, an autoencoding recurrent neural network is used to extract semantic feature vectors from speech data. The input data and output data of the autoencoding recurrent neural network are both the above-mentioned speech data. When training the network, there is no need to label the speech data. A large amount of speech data is collected in advance as the input and output of the network. The network determines the parameters through self-learning.

[0101] Alternatively, in another embodiment, other feature extraction methods may be used to obtain speech feature vectors. The first feature extraction module 401 is also used to: when receiving a voice command, obtain the speech data collected by the speech component; convert the speech data into a spectrogram according to an audio feature extraction algorithm; generate a semantic feature vector of the speech data according to a pre-trained autoencoder convolutional neural network and the spectrogram, and use the semantic feature vector as the first feature vector.

[0102] Among them, the audio feature extraction algorithm can be an MFCC (Mel Frequency Cepstrum Coefficient) algorithm or an FFT (Fast Fourier Transformation) algorithm, and the speech data is converted into a spectrogram through the audio feature extraction algorithm, and the spectrogram is used as the input data and output data of the autoencoder convolutional neural network to extract the semantic feature vector from the network. Similar to the above-mentioned autoencoder recurrent neural network, the autoencoder convolutional neural network is also an autoencoder, wherein the autoencoder convolutional neural network uses a convolutional layer to construct an autoencoder, and the output data of the autoencoder is trained to be consistent with the input data to obtain valuable information in the middle hidden layer.

[0103] The first feature extraction module 401 obtains the semantic vector feature through the above solution and uses the semantic vector feature as the first feature vector.

[0104] For another example, in one embodiment, the user instruction is a touch instruction; the first feature extraction module 401 is further used to: when receiving the touch instruction, obtain touch data collected by the touch sensor; and convert the touch data into the first feature vector.

[0105] Among them, the electronic device is provided with a touch sensor for detecting touch operations, such as a touch sensor provided under a touch screen, which can detect various touch operations and touch gestures input by the user. When a touch command is received, the touch data collected by the touch sensor is obtained, and the touch data includes the touch position and the touch track.

[0106] For another example, in one embodiment, the user instruction is a holding instruction; the first feature extraction module 401 is further used to: when receiving the holding instruction, obtain the holding data collected by the holding sensor; and convert the holding data into the first feature vector.

[0107] Among them, the frame, back cover and other positions of the electronic device are provided with a grip sensor, which can detect the grip instruction triggered by the user's grip operation. The grip data can be position information, such as position coordinates. In order to facilitate the subsequent fusion of multiple feature vectors into a feature matrix, it is necessary to predefine the preset length of the feature vector. The electronic device converts the acquired position coordinates into a vector representation of a preset length. For example, if the preset vector length is 10, the position coordinates with a length of 2 are converted into a first feature vector with a length of 10 by repeated superposition.

[0108] The second feature extraction module 402 is used to obtain current panoramic data and generate a second feature vector according to the panoramic data.

[0109] In the embodiment of the present application, the panoramic data of the electronic device includes but is not limited to the following types of data: terminal status data, user status data and sensor status data. Among them, the user status data includes the user's facial image indirectly captured by the camera of the electronic device, and the user's age, gender and other user information obtained from the user database.

[0110] The terminal operation data includes the operation mode of the user terminal in each time interval, where the operation mode includes game mode, entertainment mode, audio and video mode, etc. The operation mode of the terminal can be determined according to the type of the currently running application. The type of the currently running application can be directly obtained from the classification information of the application installation package; or, the terminal operation data can also include the remaining power of the terminal, display mode, network status, screen off / locked status, etc.

[0111] The sensor status data includes signals collected by various sensors on the electronic device. For example, the electronic device includes the following sensors: distance sensor, magnetic field sensor, light sensor, acceleration sensor, fingerprint sensor, Hall sensor, position sensor, gyroscope, inertial sensor, attitude sensor, barometer, heart rate sensor and other sensors. The sensor status data of the electronic device when receiving the user command is obtained, or the sensor status data of the electronic device for a period of time before receiving the user command is obtained. In some embodiments, the status data of some sensors can be obtained in a targeted manner.

[0112] Optionally, in one embodiment, the second feature extraction module 402 is also used to: obtain current terminal status data, user status data and sensor status data; generate terminal status features based on the terminal status data, generate user status features based on the user status data, and generate terminal scenario features based on the sensor status data; and fuse the terminal status features, the user status features and the terminal scenario features to generate the second feature vector.

[0113] In some embodiments, the second feature extraction module 402 is further used to: call the camera component to capture the user's facial image; recognize the user's facial image according to the preset convolutional neural network model to generate a user emotion tag; obtain user information, and use the user emotion tag and the user information as the user status data. The user information can be obtained from the user database, and the user information may include the user's gender, user age, user hobbies, etc.

[0114] Among them, in some embodiments, the second feature extraction module 402 is also used to: determine the program category described by the currently running application; determine the current operating mode of the terminal according to the program category, where the operating mode includes game mode, entertainment mode, and audio and video mode; and use the operating mode as terminal status data.

[0115] In summary, the terminal state feature ys1 is generated according to the terminal state data. According to the state data of the sensor, for example, the four-dimensional terminal posture features ys2 to ys5 are obtained from the magnetometer, accelerometer, and gyroscope through the Kalman filter algorithm, the air pressure feature ys6 is obtained through the data collected by the barometer, the WIFI connection status ys7 is determined through the network module, and the data collected by the position sensor is used for positioning to obtain the user's current location attributes (such as shopping malls, homes, companies, parks, etc.), and generate feature ys8; it can also be further combined with the magnetometer, accelerometer, gyroscope, barometer 10-axis information using a filtering algorithm or a principal component analysis algorithm to obtain new multidimensional data and generate the corresponding feature ys9. Generate feature ys according to the user's emotional label 10 , generate feature ys based on the user's gender, age, and hobbies11 ~ys 13 . For the non-digital features in the above features, you can use the method of establishing index numbers to convert them into digital representations. For example, for the feature of the operating mode of the current system terminal state, use the index number to represent the current state mode, such as 1 for game mode, 2 for entertainment mode, and 3 for audio and video mode. If the current operating mode is game mode, then determine that the current system state ys1 = 1. After obtaining all the digitally represented features, merge the above feature data to obtain a long vector, normalize the long vector, and obtain the second feature vector s2:

[0116] s2={ys1,ys2,…,ys m}

[0117] The feature fusion module 403 is used to fuse the first feature vector and the second feature vector to generate a user feature matrix.

[0118] The feature fusion module 403 may perform matrix superposition of the first feature vector s1 and the second feature vector s2 to generate the following user feature matrix:

[0119]

[0120] If the lengths of the first eigenvector and the second eigenvector are not equal, the shorter vector can be adjusted by zero padding. If n < m, the length of the first eigenvector s1 is extended to m by zero padding. If n > m, the length of the second eigenvector s2 is extended to n by zero padding.

[0121] Alternatively, in an optional implementation, after adjusting the first eigenvector and the second eigenvector to the same length, the feature fusion module 403 performs matrix superposition on the first eigenvector s1 and the second eigenvector s2, and then, in order to generate a richer feature vector for subsequent operations, performs a flip operation on the superimposed matrix to generate the following matrix:

[0122]

[0123] The matrices before and after the flipping operation are combined to obtain the following matrix as the user feature matrix.

[0124]

[0125] The intention prediction module 404 is used to obtain the intention label matching the user instruction according to the user feature matrix and the pre-trained intention prediction model.

[0126] In the embodiment of the present application, the intention prediction model is a classification model that characterizes the relationship between the user feature matrix and the intention label. For example, the intention prediction model can be obtained by training a classification algorithm such as a convolutional neural network, a BP neural network (Back Propagation) or an SVM (Support Vector Machine) algorithm. Taking the convolutional neural network as an example, a large amount of sample data of test users is collected, and features are extracted according to steps 101 to 103, and then the extracted features are labeled. For example, a part of users are first selected from all users as test users, and the user instructions of these users are recorded, and a first feature vector is generated; and the panoramic data when the user instruction is received is collected, and a second feature vector is generated according to the panoramic data. Then the response of the electronic device to the user instruction and the operation performed by the user based on the response are recorded. In an optional embodiment, the intention label data can be automatically generated according to the response of the electronic device to the user instruction and the operation performed by the user based on the response, or, in some other embodiments, the user feature matrix can be labeled by manual labeling.

[0127] The user feature matrix with intent labels is input into a convolutional neural network for training, wherein the structure and hyperparameters of the convolutional neural network can be pre-set by the user as needed. After training, the weight parameters of the network are determined to generate an intent prediction model.

[0128] In addition, it should be noted that in order to fully characterize user intentions, the intent label corresponding to a user feature matrix is ​​an intent label set, which contains multiple labels that can characterize user instructions, location, time, environment, status, habits and other information. These labels depict the user's current panoramic portrait.

[0129] The user feature matrix obtained by the feature fusion module 403 is input into the intention prediction model to generate an intention label that matches the user instruction.

[0130] The instruction execution module 405 is used to execute the user instruction according to the intention tag.

[0131] The intention tag obtained by the intention prediction module 404 is a tag set composed of multiple tags, which constitute a panoramic portrait corresponding to the user instruction. The instruction execution module 405 pushes the tag set to the target application while responding to the user instruction and starting the corresponding target application. In this way, when the electronic device responds to the user instruction, it not only performs a simple operation such as starting the software, but can also further perform more specific operations according to the received tag set after the target application is started.

[0132] For example, the instruction execution module 405 is also used to: determine the target application corresponding to the user instruction according to the intention tag; start the target application and send the intention tag to the target application, wherein the target application performs the corresponding operation based on the intention tag.

[0133] For example, when a user triggers a voice command "Xiaoou Xiaoou, order me food", the system's voice recognition module detects the command and obtains more data that can reflect the user's intention through panoramic perception technology, such as terminal status data, user status data, and sensor status data, etc. These data are used to supplement the context information of the command to obtain more comprehensive intention information, such as: current time, current location, scene, user eating habits, user hunger, etc. When the electronic device opens a third-party ordering application according to the command, this information is pushed to the third-party ordering application, and the third-party ordering application can recommend specific ordering locations and ordering contents to the user based on the above information.

[0134] From the above, it can be seen that the instruction execution device proposed in the embodiment of the present application, when receiving a user instruction, the first feature extraction module 401 generates a first feature vector according to the user instruction, the second feature extraction module 402 obtains the current panoramic data, and generates a second feature vector according to the panoramic data, and the feature fusion module 403 fuses the first feature vector and the second feature vector to generate a user feature matrix. Next, the intention prediction module 404 uses the user feature matrix as input data of a pre-trained intention prediction model to obtain an intention label that matches the user instruction, and the instruction execution module 405 executes the user instruction according to the intention label. This scheme supplements the context information and situation information of the user instruction by collecting panoramic data, so as to more accurately understand the user's implicit intention and thus better execute the user instruction.

[0135] The present application also provides an electronic device. The electronic device may be a smart phone, a tablet computer, or the like. Figure 7 As shown, Figure 7 The first structural diagram of the electronic device provided in the embodiment of the present application. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 is electrically connected to the memory 302.

[0136] The processor 301 is the control center of the electronic device 300. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or calling computer programs stored in the memory 302, and calling data stored in the memory 302, thereby monitoring the electronic device as a whole.

[0137] In this embodiment, the processor 301 in the electronic device 300 loads instructions corresponding to one or more computer program processes into the memory 302 according to the following steps, and the processor 301 runs the computer program stored in the memory 302 to implement various functions:

[0138] When receiving a user instruction, generating a first feature vector according to the user instruction;

[0139] Acquire current panoramic data, and generate a second feature vector according to the panoramic data;

[0140] fusing the first feature vector and the second feature vector to generate a user feature matrix;

[0141] Acquire an intent label matching the user instruction according to the user feature matrix and a pre-trained intent prediction model;

[0142] The user instruction is executed according to the intention tag.

[0143] In some embodiments, the user instruction is a voice instruction; when receiving the user instruction, when generating the first feature vector according to the user instruction, the processor 301 performs the following steps:

[0144] When receiving a voice command, obtain the voice data collected by the voice component;

[0145] A semantic feature vector of the speech data is generated according to a pre-trained autoencoding recurrent neural network, and the semantic feature vector is used as the first feature vector.

[0146] In some embodiments, the user instruction is a voice instruction; when receiving the user instruction, when generating the first feature vector according to the user instruction, the processor 301 performs the following steps:

[0147] When receiving a voice command, obtain the voice data collected by the voice component;

[0148] Converting the speech data into a spectrogram according to an audio feature extraction algorithm;

[0149] A semantic feature vector of the speech data is generated according to a pre-trained autoencoder convolutional neural network and the spectrogram, and the semantic feature vector is used as the first feature vector.

[0150] In some embodiments, the user instruction is a touch instruction; when receiving the user instruction, when generating the first feature vector according to the user instruction, the processor 301 performs the following steps:

[0151] When a touch command is received, the touch data collected by the touch sensor is acquired;

[0152] The touch data is converted into the first feature vector.

[0153] In some embodiments, the panoramic data includes terminal status data, user status data, and sensor status data; when acquiring the current panoramic data and generating the second feature vector according to the panoramic data, the processor 301 performs the following steps:

[0154] Obtain current terminal status data, user status data and sensor status data;

[0155] generating a terminal state feature according to the terminal state data, generating a user state feature according to the user state data, and generating a terminal scenario feature according to the sensor state data;

[0156] The terminal state feature, the user state feature and the terminal scenario feature are integrated to generate the second feature vector.

[0157] In some embodiments, when acquiring current user status data, the processor 301 performs the following steps:

[0158] Call the camera component to capture the user's facial image;

[0159] Recognize the user's facial image according to a preset convolutional neural network model to generate a user emotion label;

[0160] User information is obtained, and the user emotion tag and the user information are used as the user state data.

[0161] In some embodiments, when the first feature vector and the second feature vector are fused to generate a user feature matrix, the processor 301 performs the following steps:

[0162] Perform matrix superposition processing on the first eigenvector and the second eigenvector to generate a user feature matrix.

[0163] In some embodiments, when executing the user instruction according to the intention tag, the processor 301 performs the following steps:

[0164] Determining a target application corresponding to the user instruction according to the intention tag;

[0165] The target application is started, and the intent tag is sent to the target application, wherein the target application performs a corresponding operation based on the intent tag.

[0166] The memory 302 can be used to store computer programs and data. The computer programs stored in the memory 302 contain instructions that can be executed in the processor. The computer programs can form various functional modules. The processor 301 executes various functional applications and data processing by calling the computer programs stored in the memory 302.

[0167] In some embodiments, Figure 8 As shown, Figure 8 A second structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 300 further includes: a radio frequency circuit 303, a display screen 304, a control circuit 305, an input unit 306, an audio circuit 307, a sensor 308, and a power supply 309. The processor 301 is electrically connected to the radio frequency circuit 303, the display screen 304, the control circuit 305, the input unit 306, the audio circuit 307, the sensor 308, and the power supply 309, respectively.

[0168] The radio frequency circuit 303 is used to send and receive radio frequency signals to communicate with network devices or other electronic devices through wireless communication.

[0169] The display screen 304 may be used to display information input by a user or information provided to a user and various graphical user interfaces of the electronic device. These graphical user interfaces may be composed of images, texts, icons, videos and any combination thereof.

[0170] The control circuit 305 is electrically connected to the display screen 304 and is used to control the display screen 304 to display information.

[0171] The input unit 306 can be used to receive input numbers, character information or user feature information (such as fingerprints), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. The input unit 306 can include a fingerprint recognition module.

[0172] The audio circuit 307 can provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 307 includes a microphone. The microphone is electrically connected to the processor 301. The microphone is used to receive voice information input by the user.

[0173] The sensor 308 is used to collect external environment information. The sensor 308 may include one or more sensors such as an environment brightness sensor, an acceleration sensor, and a gyroscope.

[0174] The power supply 309 is used to supply power to various components of the electronic device 300. In some embodiments, the power supply 309 may be logically connected to the processor 301 through a power management system, so that the power management system can manage charging, discharging, power consumption, and other functions.

[0175] although Figure 8 Not shown, the electronic device 300 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0176] As can be seen from the above, an embodiment of the present application provides an electronic device, which can receive user instructions, such as voice instructions, touch instructions or holding instructions, etc. When receiving the user instruction, a first feature vector is generated according to the user instruction; then the current panoramic data of the electronic device is collected, such as user information, sensor status data, usage information of application programs in the electronic device, etc., and a second feature vector is generated according to the collected panoramic data. Next, the first feature vector and the second feature vector are fused to generate a user feature matrix, and an intent label corresponding to the user feature matrix is ​​generated according to the user feature matrix and a pre-trained intention prediction model, and the user instruction is executed according to the intent label. In this way, according to the panoramic data collected when receiving the instruction, the user instruction is supplemented with complete context information and information of the situational environment, so that the electronic device can more accurately understand the user's implicit intention, thereby better executing the user instruction.

[0177] An embodiment of the present application further provides a storage medium, in which a computer program is stored. When the computer program runs on a computer, the computer executes the instruction execution method described in any of the above embodiments.

[0178] It should be noted that, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0179] The above is a detailed introduction to the instruction execution method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application; at the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for executing an instruction, characterized in that: include: When receiving a user instruction, generating a first feature vector according to the user instruction; Acquire current panoramic data, and generate a second feature vector according to the panoramic data; the panoramic data includes terminal status data, user status data and sensor status data; fusing the first feature vector and the second feature vector to generate a user feature matrix; Acquire an intent label matching the user instruction according to the user feature matrix and a pre-trained intent prediction model; Execute the user instruction according to the intention tag; The step of acquiring current panoramic data and generating a second feature vector according to the panoramic data includes: Obtain current terminal status data, user status data and sensor status data; generating a terminal state feature according to the terminal state data, generating a user state feature according to the user state data, and generating a terminal scenario feature according to the sensor state data; Fusion of the terminal state feature, the user state feature and the terminal scenario feature to generate the second feature vector; The fusing the first feature vector and the second feature vector to generate a user feature matrix specifically includes: Performing matrix superposition processing on the first eigenvector and the second eigenvector to generate a user feature matrix; If the lengths of the first eigenvector and the second eigenvector are not equal, the shorter vector is adjusted by zero padding to adjust the first eigenvector and the second eigenvector to vectors of the same length; The first eigenvector and the second eigenvector after the length adjustment are matrix superimposed, and the matrix obtained by superposition is flipped, and the matrix obtained by merging the matrices before and after the flipping operation is used as the user feature matrix.

2. The instruction execution method according to claim 1, characterized in that: The user instruction is a voice instruction; when the user instruction is received, the step of generating a first feature vector according to the user instruction includes: When receiving a voice command, obtain the voice data collected by the voice component; A semantic feature vector of the speech data is generated according to a pre-trained autoencoding recurrent neural network, and the semantic feature vector is used as the first feature vector.

3. The instruction execution method according to claim 1, characterized in that: The user instruction is a voice instruction; when the user instruction is received, the step of generating a first feature vector according to the user instruction includes: When receiving a voice command, obtain the voice data collected by the voice component; Converting the speech data into a spectrogram according to an audio feature extraction algorithm; A semantic feature vector of the speech data is generated according to a pre-trained autoencoder convolutional neural network and the spectrogram, and the semantic feature vector is used as the first feature vector.

4. The instruction execution method according to claim 1, characterized in that: The user instruction is a touch instruction; when the user instruction is received, the step of generating a first feature vector according to the user instruction includes: When a touch command is received, the touch data collected by the touch sensor is acquired; The touch data is converted into the first feature vector.

5. The instruction execution method according to claim 1, characterized in that: The steps to obtain the current user status data include: Call the camera component to capture the user's facial image; Recognize the user's facial image according to a preset convolutional neural network model to generate a user emotion label; User information is obtained, and the user emotion tag and the user information are used as the user state data.

6. The instruction execution method according to any one of claims 1 to 4, characterized in that: The step of executing the user instruction according to the intention tag includes: Determining a target application corresponding to the user instruction according to the intention tag; The target application is started, and the intent tag is sent to the target application, wherein the target application performs a corresponding operation based on the intent tag.

7. The instruction execution method according to any one of claims 1 to 4, characterized in that: The step of obtaining an intent label matching the user instruction according to the user feature matrix and the pre-trained intent prediction model comprises: According to the user feature matrix and the intention prediction model obtained based on neural network training, an intention label matching the user instruction is obtained.

8. An instruction execution device, characterized in that: include: A first feature extraction module, configured to generate a first feature vector according to a user instruction when receiving the user instruction; A second feature extraction module, used to obtain current panoramic data and generate a second feature vector according to the panoramic data; the panoramic data includes terminal status data, user status data and sensor status data; A feature fusion module, used for fusing the first feature vector and the second feature vector to generate a user feature matrix; An intention prediction module, used to obtain an intention label matching the user instruction according to the user feature matrix and a pre-trained intention prediction model; An instruction execution module, used for executing the user instruction according to the intention tag; The second feature extraction module is specifically used to obtain current terminal state data, user state data and sensor state data; generate terminal state features according to the terminal state data, generate user state features according to the user state data, and generate terminal scenario features according to the sensor state data; Fusion of the terminal state feature, the user state feature and the terminal scenario feature to generate the second feature vector; The feature fusion module is specifically used to perform matrix superposition processing on the first feature vector and the second feature vector to generate a user feature matrix; wherein, if the lengths of the first feature vector and the second feature vector are not equal, the vector with the shorter length is adjusted by zero padding, and the first feature vector and the second feature vector are adjusted to vectors of the same length; the first feature vector and the second feature vector after the adjusted length are matrix superimposed, and the matrix obtained by superposition is flipped, and the matrix obtained by merging the matrices before and after the flipping operation is used as the user feature matrix.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the instruction execution method according to any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, wherein: The processor is configured to execute the instruction execution method according to any one of claims 1 to 7 by calling the computer program.

Images