Keyword identification method, device, equipment, storage medium and automobile

Abstract

The application relates to a keyword recognition method, device, equipment, storage medium and automobile, which comprises the following steps: acquiring target lip image region data, target historical feature cache data and target recent feature cache data, dividing the target lip image region data according to a channel dimension to generate second channel data; generating output data of a first target network layer through the second channel data, the target historical feature cache data and the target lip image region data; generating output data of a second target network layer through the output data of the first target network layer and the target recent feature cache data; outputting third target network layer output data through a third target network layer of a keyword recognition model; and determining a label category through the three kinds of output data, wherein the label category is used for recognizing a keyword output by a user, the keyword recognition model designed in the embodiment has a small amount of calculation, and can output a detection result in real time, quickly and relatively accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence computer vision, in particular to a keyword recognition method, device, equipment, storage medium and automobile. BACKGROUND

[0002] With the development of technology and the rise of artificial intelligence, automobiles have gradually developed towards technology and intelligence. As the main interactive entrance of the vehicle, voice recognition becomes particularly important. In some normal scenarios, voice recognition can also well recognize the user's speech, but in scenarios such as tire noise, wind noise, air conditioner noise, other passengers speaking, playing some music, and the user speaking softly, the accuracy of voice recognition will be affected. In order to improve the accuracy of keyword recognition in these scenarios and improve the user experience, image recognition is combined with voice recognition for comprehensive processing to obtain the final result.

[0003] The keyword recognition through computer vision has experienced some technical development and iteration. In the early stage, the hidden Markov model was used to quantitatively cluster the lip movement sequence feature vectors by using the K-means clustering algorithm as the input for training and recognition to achieve recognition. There are also ways to achieve it by combining convolutional neural networks and recurrent neural networks, and there are also ways to achieve it by combining three-dimensional convolutional neural networks and convolutional neural networks. However, in the first of the above three ways, the current state is only related to the previous state in the state transition process, and the problems that can be handled are relatively limited, and the calculation is relatively complex. The second implementation method has a large amount of calculation and takes a lot of time, and also has the problem of gradient disappearance. The third implementation method has a large amount of parameters and calculation, and also takes a lot of time, and the timing information extraction capability is relatively insufficient, and the recognition accuracy is not high. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a keyword recognition method, device, equipment, storage medium and automobile, which solves the problems that the hidden Markov model has limited processing problems and relatively complex calculation when recognizing keywords through computer vision in the prior art, and the problems that the combination of convolutional neural networks and recurrent neural networks has a large amount of calculation, takes a lot of time, and also has the problem of gradient disappearance, and the combination of three-dimensional convolutional neural networks and convolutional neural networks has a large amount of parameters and calculation, takes a lot of time, and the timing information extraction capability is relatively insufficient, and the recognition accuracy is not high. The specific technical solutions are as follows:

[0005] According to a first aspect of the embodiments of the present application, a keyword recognition method is provided, which comprises:

[0006] Obtaining target input data of a keyword recognition model, the target input data comprising: target lip image region data, target historical feature cache data and target recent feature cache data;

[0007] Dividing the target lip image region data according to a channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model;

[0008] Performing a first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate output data of the first target network layer, the first target network layer being a specific network layer in the keyword recognition model network layer;

[0009] Performing a second target operation on the output data of the first target network layer and the target recent feature cache data to generate output data of a second target network layer in the first target network layer;

[0010] Outputting third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model;

[0011] Determining a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer, the label category being used for recognizing a keyword output by a user.

[0012] Optionally, the first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate the output data of the first target network layer comprises:

[0013] Concatenating and convoluting the second channel data and the target historical feature cache data;

[0014] Obtaining feature data corresponding to the target lip image region data through the first target network layer of the keyword recognition model;

[0015] Adding the data after concatenation and convolution to the feature data to generate the output data of the first target network layer;

[0016] The second target operation on the output data of the first target network layer and the target recent feature cache data to generate the output data of the second target network layer in the first target network layer comprises:

[0017] Concatenate the target recent feature cache data and perform feature enhancement processing to generate recent feature enhancement data;

[0018] For a second target network layer in the first target network layer, add the output data of the first target network layer and the recent feature enhancement data to generate output data of the second target network layer.

[0019] Optionally, before the target input data of the keyword recognition model is acquired, the method further includes:

[0020] Acquire a target image including a face through a face detection algorithm;

[0021] Input the target image into a deep learning model to perform lip effective detection;

[0022] If the lip of the target image is effective, acquire face key point coordinates of the target image;

[0023] Determine a lip image region corresponding to the target image through the face key point coordinates of the target image.

[0024] Optionally, the face key point coordinates of the target image include a nose tip coordinate, a left corner coordinate, and a right corner coordinate;

[0025] The determination of the lip image region corresponding to the target image through the face key point coordinates of the target image includes:

[0026] Generate a first distance through the nose tip coordinate and the left corner coordinate;

[0027] Generate a second distance through the nose tip coordinate and the right corner coordinate;

[0028] Generate a third distance through the left corner coordinate and the right corner coordinate;

[0029] Determine a height of the lip image region in the target image through the first distance, the second distance, and the third distance;

[0030] Determine a vertex coordinate of the lip image region in the target image through the height of the lip image region and the nose tip coordinate;

[0031] Determine the lip image region corresponding to the target image through the vertex coordinate of the lip image region.

[0032] Optionally, after the label category corresponding to the target lip image region data is determined through the output data of the first target network layer, the output data of the second target network layer, and the output data of the third target network layer, the method further includes:

[0033] obtaining a length of a target data queue;

[0034] in a case where it is detected that the length of the target data queue is greater than a first preset value, deleting element data at a head of the target data queue;

[0035] inserting element data corresponding to the label category into a tail of the target data queue.

[0036] Optionally, after the element data corresponding to the label category is inserted into the tail of the target data queue, the method further includes:

[0037] filtering a sub-queue from the target data queue according to a preset rule;

[0038] obtaining a confidence degree of each element data in the sub-queue;

[0039] determining a final label category through the confidence degree of the element data.

[0040] Optionally, the determining the final label category through the confidence degree of the element data includes:

[0041] deleting a maximum value and a minimum value in the confidence degree of the element data;

[0042] obtaining an average value of the confidence degree of the element data after the deletion;

[0043] in a case where it is determined that the average value is greater than a second preset value, obtaining target element data in the element data, the target element data being element data in the element data that has a number of continuous repeated occurrences greater than a third preset value;

[0044] determining the final label category through the target element data.

[0045] Optionally, before the obtaining the target input data of the keyword recognition model, the method further includes:

[0046] obtaining first color coding data of an image through a target device;

[0047] converting the first color coding data into target color coding data including a target size under a target visual angle according to preset image width-height parameters, image format parameters and image angle parameters.

[0048] According to a second aspect of the embodiment of the present application, a keyword recognition device is provided, and the device includes:

[0049] a first obtaining module, configured to obtain target input data of a keyword recognition model, the target input data including target lip image region data, target historical feature cache data and target recent feature cache data;

[0050] a first division module, configured to divide the target lip image region data according to a channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model;

[0051] a first generation module, configured to perform a first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate output data of the first target network layer, the first target network layer being a specific network layer in the keyword recognition model network layer;

[0052] a second generation module, configured to perform a second target operation on the output data of the first target network layer and the target recent feature cache data to generate output data of a second target network layer in the first target network layer;

[0053] a first output module, configured to output third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model;

[0054] a first determination module, configured to determine a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer, the label category being used to identify a keyword output by a user.

[0055] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising:

[0056] a processor;

[0057] a memory for storing instructions executable by the processor;

[0058] The processor is configured to execute the instructions to implement the keyword recognition method according to the first aspect.

[0059] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can execute the keyword recognition method according to the first aspect of the present application.

[0060] The technical solutions provided by the embodiments of the present application can have the following beneficial effects:

[0061] The application obtains target input data of a keyword recognition model, the target input data comprising: target lip image region data, target historical feature cache data and target recent feature cache data; divides the target lip image region data according to a channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model; performs a first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate output data of the first target network layer, the first target network layer being a specific network layer in the keyword recognition model network layer; performs a second target operation on the output data of the first target network layer and the target recent feature cache data to generate output data of a second target network layer in the first target network layer; outputs third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model; and determines a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer, the label category being used for recognizing a keyword output by a user. The keyword recognition model designed in the embodiment of the application determines the keyword output by the user through image recognition, can reduce misrecognition in a scene with noise, background music, other personnel interference and the like, and improves the interactive experience of the user. Moreover, the keyword recognition model designed in the application has a small amount of calculation, can output a detection result in real time, quickly and relatively accurately, and in addition, the embodiment of the application also calculates an effective lip image region in a face image, uses the effective lip image region as data input into the keyword recognition model, can reduce calculation in a case where the mouth is invisible or partially visible, and avoids output of an error result. Moreover, the embodiment of the application also judges and verifies the result output by the keyword recognition model, further increases the accuracy of keyword recognition.

[0062] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings, which are incorporated into and form part of the specification, illustrate an embodiment consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0064] Figure 1 is a flowchart of a keyword recognition method according to an exemplary embodiment;

[0065] Figure 2 is a flowchart of another keyword recognition method according to an exemplary embodiment;

[0066] Figure 3 is a flowchart of another keyword recognition method according to an exemplary embodiment;

[0067] Figure 4 is Figure 2 is a human face key point position distribution diagram in another keyword recognition method according to an example embodiment.

[0068] Figure 5 is a block diagram of a keyword recognition device according to an example embodiment.

[0069] Figure 6 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0070] The example embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, same numbers refer to same elements throughout the drawings. The following example embodiments described in the example embodiments do not represent all the implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0071] A first embodiment of the present disclosure relates to a keyword recognition method, Figure 1 is a flowchart of a keyword recognition method according to an example embodiment, as Figure 1 is shown, including the following steps:

[0072] In step 101, target input data of a keyword recognition model is acquired, and the target input data includes target lip image region data, target historical feature cache data, and target recent feature cache data.

[0073] The embodiment of the present disclosure is directed to a scenario in which the accuracy of voice recognition is affected by tire noise, wind noise, air conditioner noise, other passengers speaking, playing some music, and the user speaking in a low voice, etc. during driving, and image recognition needs to be combined to recognize the keyword output by the user. Therefore, a lightweight keyword recognition model is designed. The input data of the model has three types. One is target lip image region data. Because the keyword spoken by the user needs to be recognized, only the target lip image region data of the image needs to be extracted. In addition, because the image obtained when the user speaks is not only one frame, it should be continuous multiple frames of images. Therefore, target historical feature cache data and target recent feature cache data are also obtained to better recognize the features corresponding to the target lip image region data. The target historical feature cache data and the target recent feature cache data are both multiple feature cache data obtained closest in time.

[0074] It should be noted that before obtaining the target input data of the keyword recognition model, the target input data also needs to be processed, including obtaining initial image data through the vehicle-mounted camera, at this time the image data is the first color coding data, that is, YUV data, and then setting the size, format and angle of the desired image in the Android system, so as to convert the first color coding data into target color coding data including a target size under a target view angle, and the specific implementation steps include:

[0075] obtaining first color coding data of the image through the target device;

[0076] converting the first color coding data into target color coding data including a target size under a target view angle according to preset image width-height parameters, image format parameters and image angle parameters.

[0077] The target view angle is set according to the angle of obtaining the image, because the image obtained by the vehicle-mounted camera is not all obtained from the front, but also obtained from the side, at this time, in order to display the obtained image at a suitable angle, the angle of the image needs to be adjusted, in addition, the preset image parameters can be set by oneself, for example, the preset image width is set to 1920, the height is set to 1080, the image format is RGB, and the image angle is left deflection 10°, according to these settings, a target color coding data including a target size under a target view angle with a width of 1920, a height of 1080, a format of RGB and an angle of left deflection is obtained.

[0078] In step 102, the target lip image region data is divided according to the channel dimension to generate first channel data and second channel data for the first target network layer of the keyword recognition model.

[0079] The keyword recognition model of the embodiment of the application is provided with a plurality of feature extraction network layers, the first target network layer refers to a specific network layer in the network layer of the keyword recognition model, for example, the keyword recognition model is provided with 19 layers of feature extraction network layers, the fourth layer, the seventh layer, the fourteenth layer and the seventeenth layer are set as the first target network layer, it should be noted that the setting of the feature extraction network layer can be changed according to the demand, such as setting 17 layers or 21 layers, the selection of the first target network layer can also be changed according to the demand, and the application does not make specific limitation here.

[0080] The target lip image region data is input into any first target network layer, the first target network layer is used for performing a feature-related operation, feature data of the target lip image region data in the network layer is obtained, and then the feature data is divided according to a channel dimension to generate different channel data; for example, the target lip image region data is input into the fourth layer to obtain feature data of the target lip image region data in the fourth layer, and then the feature data is divided into a first eighth channel and a second seventh eighth channel to obtain first channel data and second channel data; it should be noted that a corresponding number of first channel data can be obtained according to the number of the first target network layers, the first channel data is used as second output of the keyword recognition model, and after output, the first channel data is sequentially added to new historical feature cache data to obtain updated historical feature cache data.

[0081] In step 103, the second channel data, the target historical feature cache data and the target lip image region data are subjected to first target operation to generate output data of the first target network layer.

[0082] The first target operation in the embodiment of the application includes splicing, convolution, addition and the like, and the specific implementation steps of generating the output data of the first target network layer by performing the first target operation on the second channel data, the target historical feature cache data and the target lip image region data include:

[0083] The second channel data and the target historical feature cache data are subjected to splicing and convolution operation.

[0084] The feature data corresponding to the target lip image region data is obtained through the first target network layer of the keyword recognition model.

[0085] The data after splicing and convolution is added to the feature data to generate the output data of the first target network layer.

[0086] It should be noted that there are multiple first target network layers, and the first target operation needs to be performed in each first target network layer.

[0087] In step 104, for the second target network layer in the first target network layer, the output data of the first target network layer and the target recent feature cache data are subjected to second target operation to generate output data of the second target network layer.

[0088] The embodiment of the present application further particularly sets a second target network layer in the first target network layer, and in the second target network layer, in addition to the first target operation, a second target operation is additionally performed on the target recent feature cache data, wherein the second target operation refers to superimposing the output data of the first target network layer and the target recent feature cache data, and specifically, for the second target network layer in the first target network layer, the output data of the first target network layer and the target recent feature cache data are subjected to the second target operation to generate the output data of the second target network layer, and the steps include:

[0089] The target recent feature cache data is spliced and subjected to feature enhancement processing to generate recent feature enhancement data.

[0090] For the second target network layer in the first target network layer, the output data of the first target network layer and the recent feature enhancement data are added to generate the output data of the second target network layer.

[0091] For example, the second target network layer in the first target network layer is set as the fourth layer, four feature data sequentially taken from the recent feature cache data are respectively named as lf1, lf2, lf3 and lf4, lf1 of 0-1 / 4 channels, lf2 of 1 / 4-2 / 4 channels, lf3 of 2 / 4-3 / 4 channels and lf4 of 3 / 4-1 channels are taken, and after being spliced and subjected to feature enhancement processing, recent feature enhancement data lf is obtained. When the fourth layer is calculated, the output data of the first target network layer and the recent feature enhancement data lf are added to obtain the output data of the fourth layer.

[0092] It should be noted that the second target network layer can be a network layer in the first target network layer, and can also be a network layer other than the first target network layer. For the network layer other than the first target network layer, when the target recent feature cache data is superimposed, the target lip image region data is subjected to feature-related operations through the network layer to obtain feature data, and then the feature data of the network layer and the target recent feature cache data are superimposed to obtain corresponding output data as the output data of the second target network layer. For example, the second target network layer is set as the third layer, four feature data sequentially taken from the recent feature cache data are respectively named as lf1, lf2, lf3 and lf4, lf1 of 0-1 / 4 channels, lf2 of 1 / 4-2 / 4 channels, lf3 of 2 / 4-3 / 4 channels and lf4 of 3 / 4-1 channels are taken, and after being spliced and subjected to feature enhancement processing, recent feature enhancement data lf is obtained. The feature data corresponding to the target lip image region data is obtained through the third layer, and the feature data and the recent feature enhancement data lf are added to obtain the output data of the third layer.

[0093] In step 105, the third target network layer of the keyword recognition model outputs third target network layer output data about the target lip image region data.

[0094] The third target network layer of the embodiment of the present application refers to the network layer of the keyword recognition model except the first target network layer, if the fourth layer, the seventh layer, the fourteenth layer and the seventeenth layer are set as the first target network layer, then the first, second, third, fifth, sixth, eighth, ninth, tenth, eleventh, twelfth, thirteenth, fifteenth, sixteenth, eighteenth and nineteenth layers are the third target network layer, and the third target network layer performs the related operation of the features of the target lip image region data.

[0095] In step 106, the first target network layer output data, the second target network layer output data and the third target network layer output data are used to determine the label category corresponding to the target lip image region data, and the label category is used to identify the keyword output by the user.

[0096] The embodiment of the present application obtains the first output of the entire keyword recognition model after the calculation of all the network layers, and the label category output by the target lip image region data can be determined through the first output, because the keyword recognition in the field of computer vision belongs to a special application of lip language recognition, and only the specific word spoken by the user needs to be detected whether it belongs to the labeled category, so the keyword output by the user can be identified through the label category.

[0097] It should be noted that the input of the keyword recognition model in the embodiment of the present application includes three kinds of data, and the output also includes three kinds of data, the first output is the data of the label category corresponding to the target lip image region data, the second output is the sum of all the first channel data of the first target network layer, and the third output is the sum of the feature data of the second target network layer after deleting the head data of the current target nearest feature cache queue and adding the feature data of the second target network layer, which is output as the new nearest feature cache data (it can be known that there is repeated data in the nearest feature cache data), wherein the feature data of the second target network layer is obtained by performing the related operation of the features of the target lip image region data through the second network layer.

[0098] The application obtains target input data of a keyword recognition model, the target input data comprising: target lip image region data, target historical feature cache data and target recent feature cache data; divides the target lip image region data according to a channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model; performs a first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate output data of the first target network layer, the first target network layer being a specific network layer in the keyword recognition model network layer; performs a second target operation on the output data of the first target network layer and the target recent feature cache data to generate output data of a second target network layer in the first target network layer; outputs third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model; and determines a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer, the label category being used to recognize a keyword output by a user. The keyword recognition model designed in the embodiment of the application determines the keyword output by the user through image recognition, can reduce misrecognition in scenes such as noise, background music and other personnel interference, improves the interactive experience of the user, and has less calculation amount, can output a detection result in real time, quickly and relatively accurately.

[0099] The second embodiment of the present application relates to a keyword recognition method, Figure 2 is a flowchart of another keyword recognition method according to an exemplary embodiment, as Figure 2 shown, comprising the following steps:

[0100] Step 201: obtaining a target image comprising a face through a face detection algorithm.

[0101] In the embodiment of the present application, the keyword output by the user is determined by combining image recognition, so when selecting the data of the input keyword recognition model, an image including a face is selected, and the action of the user's mouth is also recognized, so the lip region image in the image including the face is also recognized. In the embodiment of the present application, the effective lip image region in the face image is calculated and used as the data of the input keyword recognition model, which can reduce the calculation in the case where the mouth is invisible or partially visible, and avoid outputting incorrect results. Based on this, the embodiment of the present application first screens out the image including the face from the acquired image through a face detection algorithm. It should be noted that the face detection algorithm includes but is not limited to the MTCNN (Multi-task Cascaded Convolutional Networks) algorithm, the Dlib face detection algorithm, and the LBPH (Local Binary Pattern Histogram) algorithm.

[0102] Step 202: input the target image into the deep learning model for lip effective detection.

[0103] The target image in the embodiment of the present application is an image including a face, but since people are not always stationary, some lip images in the acquired face target image may be blurred, small, or have an angle deviation. The recognition result of these problematic lip images has a large error with the actual result, so the lip effective image needs to be screened out first. In the embodiment of the present application, a trained deep learning model is set, which can input the target image into the deep learning model for lip effective detection.

[0104] Step 203: if the lip of the target image is effective, acquire the face key point coordinates of the target image.

[0105] In the embodiment of the present application, after determining that the lip of the target image is effective, the face key point coordinates of the target image can be acquired through a face key point algorithm. It should be noted that the face key point coordinates can be pre-set to acquire the coordinates of which position, such as shown in the following table. Figure 4 As shown in the table, the positions of the face key point coordinates acquired through the face key point algorithm include: 1. left eye center, 2. right eye center, 3. nose tip, 4. left corner of the mouth, and 5. right corner of the mouth.

[0106] It should be noted that the face detection, lip effective detection, and face key point detection can be set as three models for acquisition, which can be two models of a face detection model, a lip effective detection model, and a key point detection model, or can be set in one model, and the final result can be acquired through one model. The present application does not make a specific limitation here.

[0107] In step 204, the lip image region corresponding to the target image is determined through the face key point coordinates of the target image.

[0108] After obtaining the face key point coordinates of the target image, the vertex coordinates of the lip image region can be calculated through the coordinates, and the lip image region can be determined through the vertex coordinates. The specific calculation steps include:

[0109] A first distance is generated through the nose tip coordinate and the left corner coordinate;

[0110] A second distance is generated through the nose tip coordinate and the right corner coordinate;

[0111] A third distance is generated through the left corner coordinate and the right corner coordinate;

[0112] The height of the lip image region in the target image is determined through the first distance, the second distance and the third distance;

[0113] The vertex coordinates of the lip image region in the target image are determined through the height of the lip image region and the nose tip coordinate;

[0114] The lip image region corresponding to the target image is determined through the vertex coordinates of the lip image region.

[0115] For example, the key point 3 nose tip coordinate is set as (x3, y3), the key point 4 left corner coordinate is set as (x4, y4), and the key point 5 right corner coordinate is set as (x5, y5). At this time, the first distance is d 34 The distance between the key point 3 and the key point 4 is realized through formula (1):

[0116]

[0117] The second distance is d 35 The distance between the key point 3 and the key point 5 is realized through formula (2):

[0118]

[0119] The third distance is d 45 The distance between the key point 4 and the key point 5 is realized through formula (3):

[0120]

[0121] The height size of the lip image region in the target image is realized through formula (4), (5) and (6):

[0122]

[0123]

[0124]

[0125] The top point coordinates of the lip image region in the target image (including the left upper corner top point coordinates (box_x1, box_y1) and the right lower corner top point coordinates (box_x2, box_y2) of the lip image region) are determined through the height of the lip image region and the nose tip coordinates, which are realized through formulas (7), (8), (9), and (10):

[0126]

[0127] box_y1 = y3 (8)

[0128]

[0129] box_y2 = y3 + size (10)

[0130] After the top point coordinates of the lip image region are determined, the position and size of the lip image region corresponding to the target image can be obtained.

[0131] In step 205, target input data of the keyword recognition model is obtained, and the target input data includes target lip image region data, target historical feature cache data, and target recent feature cache data.

[0132] In step 206, the target lip image region data is divided according to the channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model.

[0133] In step 207, the second channel data, the target historical feature cache data, and the target lip image region data are subjected to a first target operation to generate output data of the first target network layer.

[0134] In step 208, the output data of the first target network layer and the target recent feature cache data are subjected to a second target operation to generate output data of a second target network layer in the first target network layer.

[0135] In step 209, third target network layer output data about the target lip image region data is output through a third target network layer of the keyword recognition model.

[0136] In step 210, a label category corresponding to the target lip image region data is determined through the output data of the first target network layer, the output data of the second target network layer, and the output data of the third target network layer, and the label category is used to recognize the keyword output by the user.

[0137] It should be noted that in the embodiments of the present application, steps 205-210 are discussed in the preamble and will not be repeated here.

[0138] The keyword recognition model designed in the embodiments of the present application determines the keyword output by the user through image recognition, which can reduce misrecognition in scenes with noise, background music, other personnel interference, etc., and improve the user's interactive experience. Moreover, the keyword recognition model designed in the present application has less calculation amount, and can output detection results in real time, quickly and relatively accurately. In addition, in the embodiments of the present application, the effective lip image area in the face image is calculated and used as the data input to the keyword recognition model, which can reduce the calculation in the case where the mouth is not visible or partially visible, and avoid outputting incorrect results.

[0139] It should be noted that in the embodiments of the present application, the image is input to the face detection and key point detection model, and the corresponding lip image area of the target image is output after inference and algorithm processing. Moreover, the data of the corresponding lip image area of the target image is input to the keyword recognition model, and the label category is output after inference and algorithm processing. Finally, in the process of displaying the keyword result on the mobile terminal, the input and output calculation of the network model involved is different from that of the traditional network model, so the training method is also special. In order to obtain the trained model, a network model training method is designed in the embodiments of the present application, as follows:

[0140] The batch_size is set to 120, the number of extracted frames segments_num is set to 4, the initial learning rate lr is set to 0.001, and the SGD optimizer is used. In each data acquisition iteration of each epoch, the model is inferred and calculated 4 times, and the first input is the data corresponding to the index of the total input, and the first output is the data corresponding to the index of the total output. Meanwhile, the present application designs a loss function L for this network model training method w The formulas (11), (12) and (13) can be used to calculate:

[0141]

[0142] Wherein, s is segments_num, i.e. the number of extracted picture frames, s∈[4,32], which can be generally set to 4, 8, 16, 32, and here s is set to 4; is an inflation factor, corresponding to which can be set to 2, 1, 0.5, 0.25, and here β is an initial offset factor, set to β=0.2. The input x is an integer between 0 and s, and f(x) outputs the corresponding weight,

[0143] For the cross-entropy loss function, in a multi-classification problem, if there are N samples in total, K label values, y is the real label, and the probability of the ith sample being predicted as the kth label value is p i,k , the initial loss function L ce is as follows:

[0144]

[0145] After the above calculation, the final loss function L w is as follows:

[0146] L w = f(x) * L ce (13)

[0147] Each time the output is calculated, it is input into the loss function to calculate the update, after the set epoch training, the output final model is saved, it needs to be explained that the batch_size, the number of segments segments_num, the learning rate lr, the inflation factor , the offset factor β and other parameters involved in the above are not particularly limited, and are also applicable to other suitable values. At the same time, it is also applicable to some formulas based on formula (11) and formula (12).

[0148] The third embodiment of the present application relates to a keyword recognition method, Figure 3 is a flow chart of another keyword recognition method according to an exemplary embodiment, as Figure 3 shown, comprising the following steps:

[0149] Step 301, obtaining target input data of a keyword recognition model, the target input data comprising: target lip image region data, target historical feature cache data and target recent feature cache data.

[0150] Step 302, for the first target network layer of the keyword recognition model, the target lip image region data is divided according to the channel dimension to generate first channel data and second channel data.

[0151] Step 303, performing a first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate output data of the first target network layer.

[0152] Step 304, for the second target network layer in the first target network layer, performing a second target operation on the output data of the first target network layer and the target recent feature cache data to generate output data of the second target network layer.

[0153] Step 305, outputting third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model.

[0154] Step 306, determining a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer, and the label category is used for recognizing the keyword output by the user.

[0155] It should be noted that the steps 301-306 are described in the foregoing, and will not be described here again.

[0156] Step 307, obtaining the length of the target data queue.

[0157] Step 308, in the case that the length of the target data queue is greater than a first preset value, deleting the element data at the head of the target data queue and inserting the element data corresponding to the label category into the tail of the target data queue.

[0158] Step 309, inserting the element data corresponding to the label category into the tail of the target data queue.

[0159] After obtaining the keyword recognition model output result, the embodiment of the present application caches the result to the target data queue, and outputs the final result after processing by the determination algorithm designed by the embodiment of the present application. However, the length of the target data queue has a maximum limit, and when the maximum limit (the first preset value) is exceeded, part of the historical data needs to be cleared to save the newly cached data. Therefore, the length of the current target data queue is obtained, and when the length is greater than the threshold of the maximum limit, the element data at the head is deleted according to the first-in first-out principle, and the newly cached data is inserted into the tail of the target data queue.

[0160] Step 310, screening a sub-queue from the target data queue according to a preset rule.

[0161] The preset rule in the embodiment of the present application refers to a data queue in which the number of continuous repeated same values is greater than 8, and the 8 can also be replaced according to actual needs, which is not limited in the present application.

[0162] Step 311, obtaining the confidence of each element data in the sub-queue.

[0163] Step 312, determining the final label category through the confidence of the element data.

[0164] The step of determining the final label category through the confidence of the element data in the embodiment of the present application includes:

[0165] Delete the maximum value and the minimum value in the confidence of the element data;

[0166] Obtain the average value of the confidence of the element data after deletion;

[0167] In a case where it is determined that the average value is greater than the second preset value, obtain target element data in the element data, the target element data being element data in which the number of consecutive repeated occurrences is greater than a third preset value;

[0168] Determine the final label category through the target element data.

[0169] The second preset value is set according to actual conditions and can be 0.8, 0.7 or 0.88, which is not limited in the present application.

[0170] The keyword recognition model designed in the embodiment of the present application determines the keyword output by the user through image recognition, which can reduce misrecognition in scenes such as noise, background music and other personnel interference, improve the interactive experience of the user, and the keyword recognition model designed in the present application has less calculation amount, can output detection results in real time, quickly and relatively accurately, in addition, the embodiment of the present application further judges and verifies the results output by the keyword recognition model, further increasing the accuracy of keyword recognition.

[0171] The fourth embodiment of the present application relates to a keyword recognition device, Figure 5 is a flow chart of a keyword recognition device according to an example embodiment, as Figure 5 shown, comprising the following steps:

[0172] The first acquisition module 401 is configured to acquire target input data of the keyword recognition model, and the target input data includes target lip image region data, target historical feature cache data and target recent feature cache data.

[0173] The first division module 402 is configured to divide the target lip image region data according to the channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model.

[0174] The first generation module 403 is configured to perform first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate output data of the first target network layer, and the first target network layer refers to a specific network layer in the network layer of the keyword recognition model.

[0175] The second generation module 404 is configured to perform second target operation on the output data of the first target network layer and the target recent feature cache data to generate output data of a second target network layer for the second target network layer in the first target network layer.

[0176] The first output module 405 is configured to output third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model.

[0177] The first determination module 406 is configured to determine a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer, and the output data of the third target network layer, and the label category is used to identify the keyword output by the user.

[0178] Optionally, the first generation module 403 further includes:

[0179] The concatenation and convolution sub-module is configured to perform concatenation and convolution operations on the second channel data and the target historical feature cache data.

[0180] The first generation sub-module is configured to add the data after the concatenation and convolution to the target lip image region data to generate the output data of the first target network layer.

[0181] The second generation module 404 further includes:

[0182] The second generation sub-module is configured to perform feature enhancement processing on the target recent feature cache data after concatenation to generate recent feature enhancement data.

[0183] The third generation sub-module is configured to add the output data of the first target network layer to the recent feature enhancement data to generate the output data of the second target network layer for the second target network layer in the first target network layer.

[0184] Optionally, the keyword recognition device further includes:

[0185] The second acquisition module is configured to acquire the target image including the face through a face detection algorithm.

[0186] The effective detection module is configured to input the target image into a deep learning model for lip effective detection.

[0187] The third acquisition module is configured to acquire face key point coordinates of the target image if the lip of the target image is effective.

[0188] The second determination module is configured to determine a lip image region corresponding to the target image through the face key point coordinates of the target image.

[0189] Optionally, the face key point coordinates of the target image include a nose tip coordinate, a left corner coordinate, and a right corner coordinate.

[0190] The second determination module further includes:

[0191] The fourth generation sub-module is configured to generate a first distance by the tip-of-nose coordinate and the left corner-of-mouth coordinate.

[0192] The fifth generation sub-module is configured to generate a second distance by the tip-of-nose coordinate and the right corner-of-mouth coordinate.

[0193] The sixth generation sub-module is configured to generate a third distance by the left corner-of-mouth coordinate and the right corner-of-mouth coordinate.

[0194] The first determination sub-module is configured to determine a height of the lip image region in the target image by the first distance, the second distance and the third distance.

[0195] The second determination sub-module is configured to determine a vertex coordinate of the lip image region in the target image by the height of the lip image region and the tip-of-nose coordinate.

[0196] The third determination sub-module is configured to determine the lip image region corresponding to the target image by the vertex coordinate of the lip image region.

[0197] Optionally, the keyword recognition device further comprises:

[0198] The fourth acquisition module is configured to acquire a length of the target data queue.

[0199] The deletion module is configured to delete an element data at a head of the target data queue when it is detected that the length of the target data queue is greater than a first preset value.

[0200] The insertion module is configured to insert the element data corresponding to the label category into a tail of the target data queue.

[0201] The screening module is configured to screen a sub-queue from the target data queue according to a preset rule.

[0202] The fifth acquisition module is configured to acquire a confidence degree of each element data in the sub-queue.

[0203] The third determination module is configured to determine a final label category by the confidence degree of the element data.

[0204] Optionally, the third determination module further comprises:

[0205] The deletion sub-module is configured to delete a maximum value and a minimum value in the confidence degree of the element data.

[0206] The first acquisition sub-module is configured to acquire an average value of the confidence degree of the element data after the deletion.

[0207] The second acquisition sub-module is configured to acquire a target element data in the element data when it is determined that the average value is greater than a second preset value, the target element data being an element data in the element data which appears continuously and repeatedly more than a third preset value of times.

[0208] The fourth determining sub-module is configured to determine a final label category by using the target element data.

[0209] Optionally, the keyword recognition device further comprises:

[0210] The sixth obtaining module is configured to obtain first color coding data of the image by using the target device.

[0211] The conversion module is configured to convert the first color coding data into target color coding data including a target size under a target visual angle according to preset image width-height parameters, image format parameters and image angle parameters.

[0212] The application obtains target input data of a keyword recognition model, the target input data comprising target lip image region data, target historical feature cache data and target recent feature cache data; the target lip image region data is divided according to a channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model; the second channel data, the target historical feature cache data and the target lip image region data are subjected to a first target operation to generate output data of the first target network layer, the first target network layer being a specific network layer in the keyword recognition model network layer; the output data of the first target network layer and the target recent feature cache data are subjected to a second target operation to generate output data of a second target network layer for the second target network layer in the first target network layer; third target network layer output data about the target lip image region data is output by a third target network layer of the keyword recognition model; the label category corresponding to the target lip image region data is determined by the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer, the label category being used to recognize the keyword output by the user, the keyword recognition model designed in the application embodiment determines the keyword output by the user through image recognition, which can reduce misrecognition in the scene of noise, background music, other personnel interference and the like, and improve the interactive experience of the user, and the keyword recognition model designed in the application has a small amount of calculation, and can output the detection result in real time, quickly and relatively accurately, in addition, the effective lip image region in the face image is calculated in the application embodiment, which is used as the data input to the keyword recognition model, and the calculation in the case of invisible or partially visible mouth can be reduced to avoid output of an error result, and the result output by the keyword recognition model is determined and verified in the application embodiment, which further increases the accuracy of keyword recognition.

[0213] As to the device in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in details in the embodiments relating to the method, and thus will not be described in details here.

[0214] A fifth embodiment of the present application relates to an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement any of the keyword recognition methods.

[0215] Figure 6 is a block diagram of an electronic device 500 according to an exemplary embodiment. The electronic device 500 can be a mobile phone, a computer, a digital broadcasting terminal, a message transmitting / receiving device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0216] Referring to Figure 6 The electronic device 500 can include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0217] The processing component 502 usually controls overall operations of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 can include one or more processors 520 to execute instructions to complete all or part of steps of the above-described methods. In addition, the processing component 502 can include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0218] The memory 504 is configured to store various types of data to support operations of the device 500. Examples of these data include instructions for any application or method operating on the device 500, contact data, phonebook data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0219] The power supply component 506 supplies power for various components of the electronic device 500. The power supply component 506 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing power for the electronic device 500.

[0220] The multimedia component 508 includes a screen to provide an output interface between the electronic device 500 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and intensity of the touching or sliding action. In some embodiments, the multimedia component 508 includes a front camera and / or a back camera. When the electronic device 500 is in an operating mode, such as a camera mode or a video mode, the front camera and / or the back camera can receive external multimedia data. Each of the front camera and the back camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0221] The audio component 510 is configured to output and / or input an audio signal. For example, the audio component 510 includes a microphone (MIC) to receive an external audio signal when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further includes a speaker to output an audio signal.

[0222] The input / output interface 512 provides an interface between the processing component 502 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0223] The sensor component 514 includes one or more sensors to provide various state assessments for the electronic device 500. For example, the sensor component 514 can detect an open / closed state of the electronic device 500, relative positioning of components, such as a display and a keypad of the electronic device 500, a change in position of the electronic device 500 or a component of the electronic device 500, presence or absence of user contact with the electronic device 500, orientation or acceleration / deceleration of the electronic device 500, and a temperature change of the electronic device 500. The sensor component 514 can include a proximity sensor to detect presence of an object within a proximity range of the electronic device 500 without any physical contact. The sensor component 514 can further include a light sensor, such as a CMOS or CCD image sensor, to use in an imaging application. In some embodiments, the sensor component 514 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0224] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a cellular network standard, such as 2G, 3G, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives broadcast signals or broadcast-related information from external broadcast management systems via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0225] In an exemplary embodiment, the electronic device 500 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0226] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 504 including instructions, is also provided, which can be executed by the processor 520 of the electronic device 500 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0227] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims. Other variations of the application will be apparent to those skilled in the art and can be derived from the disclosure without departing from the scope of the application. The specification and examples are accordingly to be regarded as illustrative in nature and not restrictive.

[0228] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unitary claim, several devices or means can be listed, comprising means which can be implemented by one and the same item of hardware. The use of the words 'first','second', and 'third', etc. do not imply any ordering, but rather are used for identification purposes only. The use of any of the following terms in the claims is neither meant to limit the scope nor to imply several means for performing the steps, examples, or embodiments of the present application, but rather is considered non-limiting, and exemplary. These terms are just used to define means plus function claims. The indefinite articles 'a' and 'an' preceding an element or clause are used herein to mean zero or more than one of that element or clause. The implementation of a specific aspect can also involve the implementation of a specific aspect to which that specific aspect is incidental, interactive, or related. The application is not restricted to the details given in the description and / or shown in the drawings. Other variants and embodiments are possible within the scope of the application. The scope of the application is the scope of the claims.

Claims

1. A keyword spotting method, characterized by, The method comprises: obtaining target input data of a keyword recognition model, the target input data comprising: target lip image region data, target historical feature cache data and target recent feature cache data; dividing the target lip image region data according to a channel dimension to generate first channel data and second channel data for a first target network layer of the keyword recognition model; performing a first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate output data of the first target network layer, the first target network layer being a specific network layer in the keyword recognition model network layer; performing a second target operation on the output data of the first target network layer and the target recent feature cache data to generate output data of a second target network layer in the first target network layer; outputting third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model; determining a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer, the label category being used for recognizing a keyword output by a user.

2. The method of claim 1, wherein, The first target operation on the second channel data, the target historical feature cache data and the target lip image region data to generate the output data of the first target network layer comprises: splicing and convolution operation on the second channel data and the target historical feature cache data; obtaining feature data corresponding to the target lip image region data through the first target network layer of the keyword recognition model; adding the spliced and convoluted data and the feature data to generate the output data of the first target network layer; The second target operation on the output data of the first target network layer and the target recent feature cache data to generate the output data of the second target network layer in the first target network layer comprises: performing feature enhancement processing on the target recent feature cache data after splicing to generate recent feature enhancement data; adding the output data of the first target network layer and the recent feature enhancement data to generate the output data of the second target network layer in the first target network layer.

3. The method of claim 1, wherein, Before the target input data of the keyword recognition model is obtained, the method further comprises: obtaining a target image comprising a face through a face detection algorithm; inputting the target image into a deep learning model for lip effective detection; if the lip of the target image is effective, obtaining face key point coordinates of the target image; determining a lip image region corresponding to the target image through the face key point coordinates of the target image.

4. The method of claim 3, wherein, The face key point coordinates of the target image comprise: a nose tip coordinate, a left corner coordinate and a right corner coordinate; The determination of the lip image region corresponding to the target image through the face key point coordinates of the target image comprises: generate a first distance through the nose tip coordinate and the left corner of the mouth coordinate; generate a second distance through the nose tip coordinate and the right corner of the mouth coordinate; generate a third distance through the left corner of the mouth coordinate and the right corner of the mouth coordinate; determine the height of the lip image region in the target image through the first distance, the second distance and the third distance; determine the vertex coordinate of the lip image region in the target image through the height of the lip image region and the nose tip coordinate; determine the target lip image region corresponding to the target image through the vertex coordinate of the lip image region.

5. The method of claim 1, wherein, After the output data of the first target network layer, the output data of the second target network layer and the output data of the third target network layer are used to determine the label category corresponding to the target lip image region data, the method further comprises: acquire the length of the target data queue; in the case that the length of the target data queue is greater than the first preset value, delete the element data at the head of the target data queue; insert the element data corresponding to the label category into the tail of the target data queue.

6. The method of claim 5, wherein, After the element data corresponding to the label category is inserted into the tail of the target data queue, the method further comprises: filter a sub-queue from the target data queue according to a preset rule; acquire the confidence of each element data in the sub-queue; determine the final label category through the confidence of the element data.

7. The method of claim 6, wherein, The determination of the final label category through the confidence of the element data comprises: delete the maximum value and the minimum value in the confidence of the element data; acquire the average value of the confidence of the element data after deletion; in the case that the average value is greater than the second preset value, acquire the target element data in the element data, the target element data being the element data that appears continuously and repeatedly more than the third preset value in the element data; determine the final label category through the target element data.

8. The method of claim 1, wherein, Before the target input data of the keyword recognition model is acquired, the method further comprises: acquire first color coding data of an image through a target device; convert the first color coding data into target color coding data including a target size under a target visual angle according to preset image width and height parameters, image format parameters and image angle parameters.

9. A keyword spotting apparatus characterized by comprising: comprises: a first acquisition module, configured to acquire target input data of a keyword recognition model, the target input data comprising: target lip image region data, target historical feature cache data and target recent feature cache data; a first division module, configured to divide the target lip image region data according to a channel dimension for a first target network layer of the keyword recognition model, to generate first channel data and second channel data; a first generation module, configured to perform a first target operation on the second channel data, the target historical feature cache data and the target lip image region data, to generate output data of the first target network layer, the first target network layer being a specific network layer in the keyword recognition model network layer; a second generation module, configured to perform a second target operation on the output data of the first target network layer and the target recent feature cache data for a second target network layer in the first target network layer, to generate output data of the second target network layer; a first output module, configured to output third target network layer output data about the target lip image region data through a third target network layer of the keyword recognition model; a first determination module, configured to determine a label category corresponding to the target lip image region data through the output data of the first target network layer, the output data of the second target network layer, and the output data of the third target network layer, the label category being used to recognize a keyword output by a user.

10. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the keyword recognition method according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal is enabled to perform the keyword recognition method according to any one of claims 1 to 8.

12. A vehicle characterized by comprising: comprise the keyword recognition device according to claim 9.

Images