Image processing method and model generation method, device, vehicle, and storage medium

Abstract

The embodiments of the present application disclose an image processing method and a model generation method, device, vehicle, and storage medium, wherein the image processing method includes: obtaining an image frame sequence containing a mouth object; extracting mouth key point features for each image frame in the image frame sequence to obtain mouth key point features for each image frame; generating syllable classification features based on the mouth key point features of multiple image frames in the image frame sequence; wherein the syllable classification features represent the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence; and determining keywords that match the syllable classification features in a preset keyword library.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of information technology, and in particular to an image processing method and a model generation method, device, vehicle, and storage medium. Background Art

[0002] Lip reading recognition technology uses computer vision technology to identify faces from video images, extracting the changing features of the mouth area, and thus identifying the corresponding text content in the video. However, the accuracy of lip reading recognition in related technologies is relatively low. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide at least one image processing method and model generation method, device, vehicle, and storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] On the one hand, an embodiment of the present application provides an image processing method, which includes: obtaining an image frame sequence containing a mouth object; extracting mouth key point features for each image frame in the image frame sequence to obtain mouth key point features of each image frame; generating syllable classification features based on the mouth key point features of multiple image frames in the image frame sequence; wherein the syllable classification features represent the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence; and determining keywords that match the syllable classification features in a preset keyword library.

[0006] In some embodiments, extracting mouth key point features from each image frame in the image frame sequence to obtain mouth key point features for each image frame includes: determining position information of at least two mouth key points of the mouth object in each image frame; and determining, for each image frame in the image frame sequence, the mouth key point features corresponding to the image frame based on the position information of the mouth key points in the image frame and its adjacent frames. Compared to performing lip reading recognition using a mouth area image sequence obtained by cropping a facial image, the embodiment of the present application uses the position information of multiple mouth key points in multiple image frames to obtain mouth key point features, so that the mouth key point features can represent the changes in the mouth key points during the speaking process corresponding to the image frame sequence, thereby better extracting the changing features of the lip shape during the speaking process; in this way, using mouth key point features for lip reading recognition can improve the accuracy of lip reading recognition.

[0007] In some embodiments, the mouth key point features include inter-frame difference information and intra-frame difference information of each of the mouth key points; determining the mouth key point features corresponding to the image frame based on the position information of the mouth key points in the image frame and the adjacent frames of the image frame includes: for each of the mouth key points, determining the first height difference and / or first width difference of the mouth key point between the image frame and the adjacent frame based on the position information of the mouth key point in the image frame and the position information of the mouth key point in the adjacent image frames of the image frame, as the inter-frame difference information of the mouth key point; for each of the mouth key points, determining the intra-frame difference information of the mouth key point based on the second height difference and / or second width difference between the mouth key point in the image frame and other mouth key points of the same mouth object. In the above embodiment, the inter-frame difference information of the position information of each mouth key point in adjacent image frames and the intra-frame difference information of the position information of this mouth key point and the preset mouth key point are used to obtain the mouth key point features, thereby improving the accuracy of determining the mouth shape in each frame of the image; and the mouth key point features can also represent the inter-frame change process of the mouth key points during the speaking process corresponding to the image frame sequence; in this way, the change characteristics of the mouth shape during the speaking process can be better extracted, thereby improving the accuracy of lip reading recognition.

[0008] In some embodiments, generating syllable classification features based on the mouth key point features of multiple image frames in the image frame sequence includes: performing spatial feature extraction on the mouth key point features of each image frame to obtain the spatial features of the mouth object in each image frame; performing temporal feature extraction on the spatial features of the mouth object in multiple image frames to obtain the spatiotemporal features of the mouth object; and performing syllable classification feature extraction based on the spatiotemporal features of the mouth object to obtain the syllable classification features of the mouth object. The above embodiments support the extraction of spatiotemporal features through convolutional neural networks, which requires less computation, can reduce computing resource consumption, and lower the hardware requirements of computer devices for lip reading recognition. In particular, the use of convolutional neural networks can reduce the computing power requirements of chips, thereby enabling the image processing methods provided in the embodiments of the present application to be implemented using more lightweight chips, enabling more hardware to support the image processing methods in the lip reading recognition process of the embodiments of the present application, thereby improving the versatility of lip reading recognition. For example, computer devices such as in-vehicle computers can also implement lip reading recognition.

[0009] In some embodiments, extracting spatial features from the mouth key point features of each image frame to obtain the spatial features of the mouth object in each image frame includes: fusing inter-frame difference information and intra-frame difference information of multiple mouth key points of the mouth object to obtain inter-frame difference features and intra-frame difference features of the mouth object in each image frame; and fusing the inter-frame difference features and intra-frame difference features of the mouth object in multiple image frames to obtain the spatial features of the mouth object in each image frame. In the above embodiment, fusing the inter-frame difference information and intra-frame difference information of at least two mouth key points of the mouth object in each image frame to obtain inter-frame difference features representing inter-frame difference information between the mouth key points and intra-frame difference features representing intra-frame difference information between the mouth key points. Further fusing the inter-frame difference features and intra-frame difference features of the mouth key points in each image frame can better extract the spatial features of the mouth object in each image frame, thereby improving the accuracy of determining the lip shape in each image frame.

[0010] In some embodiments, acquiring a sequence of image frames containing a mouth object includes: acquiring a video containing the mouth object; performing lip movement recognition on the mouth object, and determining multiple video frames in which the mouth object is speaking as an image frame sequence. This allows the image frame sequence to at least cover the entire speaking process of the designated subject, and lip movement recognition reduces the number of image frames irrelevant to the speaking process in the image frame sequence. Using the image frame sequence thus obtained to perform image processing and obtain matching keywords can further improve the accuracy of lip reading recognition and reduce the computational complexity required for the image processing process of lip reading recognition.

[0011] In some embodiments, generating syllable classification features based on the mouth key point features of multiple image frames in the image frame sequence includes: using a trained syllable feature extraction network to process the mouth key point features of multiple image frames in the image frame sequence to obtain syllable classification features; and determining keywords that match the syllable classification features in a preset keyword library includes: using a trained classification network to determine keywords that match the syllable classification features in a preset keyword library. In the above embodiments, since each network in the deep learning model is learnable, the accuracy of the extracted syllable classification features and the keywords that match the syllable classification features can be improved, thereby making the keywords obtained by image processing more accurate and improving the accuracy of lip reading recognition.

[0012] In some embodiments, the syllable feature extraction network includes a spatial feature extraction subnetwork, a temporal feature extraction subnetwork and a syllable classification feature extraction subnetwork; the syllable feature extraction network in the utilization model determines the syllable classification feature based on the at least one mouth key point feature, including: utilizing the spatial feature extraction subnetwork to perform spatial feature extraction on the mouth key point features of each of the image frames to obtain the spatial features of the mouth object in each image frame; utilizing the temporal feature extraction subnetwork to perform temporal feature extraction on the spatial features of the mouth object in multiple image frames to obtain the spatiotemporal features of the mouth object; utilizing the syllable classification feature extraction subnetwork to perform classification feature extraction on each of the spatiotemporal features of the mouth object to obtain the syllable classification features of the mouth object.

[0013] In some embodiments, acquiring the image frame sequence containing the mouth object includes: performing image interpolation on the acquired original image sequence containing the mouth object to obtain the image frame sequence; or performing image interpolation on the acquired original image sequence based on the mouth key points in the acquired original image sequence containing the mouth object to obtain the image frame sequence. In this way, by performing lip reading recognition using the interpolated image frame sequence, the number of frames in the acquired original image sequence is not required during the lip reading recognition process, thereby improving the robustness of the image recognition method during the lip reading recognition process.

[0014] On the other hand, an embodiment of the present application provides a method for generating a lip reading recognition model, the method comprising: obtaining a sample image frame sequence containing a mouth object; wherein the sample image frame sequence is annotated with a keyword label; performing mouth key point feature extraction on each sample image frame in the sample image frame sequence to obtain the mouth key point features of each sample image frame; utilizing the model to be trained, generating syllable classification features based on the mouth key point features of multiple sample image frames in the sample image frame sequence, and determining keywords matching the syllable classification features in a preset keyword library; wherein the syllable classification features characterize the syllable category corresponding to the mouth shape of the mouth object in the sample image frame sequence; and updating the network parameters of the model at least once based on the determined keywords and the keyword labels to obtain a trained lip reading recognition model.

[0015] In the above-described embodiment, during the model training process, syllable-assisted learning can effectively reduce the learning difficulty of keyword recognition and classification, thereby improving the accuracy of lip reading recognition by the trained lip reading recognition model. Furthermore, because syllable classification features are determined based on mouth key point features, they can better reflect the syllables corresponding to the mouth shapes in the image frame sequence. Using syllable classification features to assist lip reading recognition makes the keywords obtained through image processing more accurate, thereby improving the accuracy of lip reading recognition. Furthermore, compared to lip reading recognition using a mouth region image sequence obtained by cropping facial images, using mouth key point features for lip reading recognition can reduce the computational effort required in the image processing process, thereby lowering the hardware requirements of the computer device executing the image processing method. Furthermore, good recognition results can be achieved for facial images with different facial shapes, textures, and other appearance information. Therefore, based on mouth key point features, the recognition capability of image categories with facial shapes and textures not considered during model training can be improved, thereby improving the generalization capability of lip reading recognition.

[0016] In some embodiments, the model includes a syllable feature extraction network and a classification network; the method of using the model to be trained to generate syllable classification features based on the mouth key point features of multiple sample image frames in the sample image frame sequence, and determining keywords that match the syllable classification features in a preset keyword library includes: using the syllable feature extraction network to generate syllable classification features based on the mouth key point features of multiple sample image frames in the sample image frame sequence; and using the classification network to determine keywords that match the syllable classification features in a preset keyword library.

[0017] In some embodiments, the syllable feature extraction network includes a spatial feature extraction subnetwork, a temporal feature extraction subnetwork, and a syllable classification feature extraction subnetwork; the syllable feature extraction network is used to generate syllable classification features based on the mouth key point features of multiple sample image frames in the sample image frame sequence, including: using the spatial feature extraction subnetwork to perform spatial feature extraction on the mouth key point features of each sample image frame respectively to obtain the spatial features of the mouth object in each sample image frame; using the temporal feature extraction subnetwork to perform sample time feature extraction on the spatial features of the mouth object in multiple sample image frames to obtain the spatiotemporal features of the mouth object; using the syllable classification feature extraction subnetwork to perform syllable classification feature extraction based on the spatiotemporal features of the mouth object to obtain the syllable classification features of the mouth object.

[0018] In another aspect, an embodiment of the present application provides an image processing device, comprising:

[0019] A first acquisition module is used to acquire a sequence of image frames containing a mouth object;

[0020] a first recognition module, configured to extract mouth key point features from each image frame in the image frame sequence to obtain mouth key point features of each image frame;

[0021] A first determining module is configured to generate a syllable classification feature based on the mouth key point features of a plurality of image frames in the image frame sequence; wherein the syllable classification feature represents the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence;

[0022] The first matching module is used to determine keywords that match the syllable classification feature in a preset keyword library.

[0023] In another aspect, an embodiment of the present application provides a device for generating a lip reading recognition model, the device comprising:

[0024] A second acquisition module is configured to acquire a sample image frame sequence containing a mouth object; wherein the sample image frame sequence is annotated with a keyword tag;

[0025] a second recognition module, configured to extract mouth key point features from each sample image frame in the sample image frame sequence to obtain mouth key point features of each sample image frame;

[0026] a second matching module, configured to generate, using the model to be trained, a syllable classification feature based on the mouth key point features of a plurality of sample image frames in the sample image frame sequence, and determine a keyword matching the syllable classification feature in a preset keyword library; wherein the syllable classification feature represents a syllable category corresponding to the mouth shape of the mouth object in the sample image frame sequence;

[0027] An updating module is used to update the network parameters of the model at least once based on the determined keywords and keyword tags to obtain a trained lip reading recognition model.

[0028] On the other hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements some or all of the steps in the above method.

[0029] In another aspect, an embodiment of the present application provides a vehicle, comprising:

[0030] an onboard camera for capturing a sequence of image frames containing a mouth object;

[0031] A vehicle computer is connected to the vehicle-mounted camera and is used to obtain an image frame sequence containing a mouth object from the vehicle-mounted camera; extract mouth key point features for each image frame in the image frame sequence to obtain mouth key point features of each image frame; generate syllable classification features based on the mouth key point features of multiple image frames in the image frame sequence; wherein the syllable classification features represent the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence; and determine keywords that match the syllable classification features in a preset keyword library.

[0032] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0033] On the other hand, an embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0034] On the other hand, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method.

[0035] In the embodiment of the present application, first, a sequence of image frames containing a mouth object in the image content is obtained; in this way, a sequence of image frames that records the change process of the mouth object when the set object speaks can be obtained; secondly, the mouth key point features are extracted for each image frame in the image frame sequence, and the mouth key point features of each image frame in the multiple image frames in the image frame sequence are obtained; compared with lip reading recognition using a mouth area image sequence obtained by cropping a facial image, lip reading recognition using mouth key point features can reduce the amount of calculation required for the image processing process, thereby reducing the hardware requirements of the computer equipment executing the image processing method; and, good recognition effects can be achieved for facial images with different appearance information such as face shapes and textures, thereby improving the generalization of lip reading recognition. Ability; secondly, generate syllable classification features based on the mouth key point features of multiple image frames in the image frame sequence, and the syllable classification features represent the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence; in this way, extract syllable classification features from the mouth key point features, and the syllable classification features can represent at least one syllable corresponding to the mouth shape of the mouth object in the image frame sequence. Using the syllable classification features to assist lip reading recognition can improve the accuracy of lip reading recognition; finally, match the matching keywords in the preset keyword library based on the syllable classification features; in this way, by representing the syllable classification features corresponding to the image frame sequence, determine the keywords of the words corresponding to the syllables according to the syllable category represented by the syllable classification features, thereby improving the accuracy of the keywords obtained by image processing. In the above scheme, the mouth key point features are extracted from the image frame in the image frame sequence to obtain the mouth key point features, and the mouth key point features are used to generate the syllable classification features corresponding to the image frame sequence, and the keywords are matched in the preset keyword library based on the syllable classification features. In this way, the amount of computation required for the image processing process of lip reading recognition can be reduced, thereby reducing the hardware requirements for computer equipment; at the same time, good recognition effects can be achieved for facial images with different face shapes, textures and other appearance information, thereby improving the generalization ability of lip reading recognition; in addition, by representing the syllable classification features corresponding to the image frame sequence, determining the keywords of the words corresponding to the syllables according to the syllable categories represented by the syllable classification features, the keywords obtained by image processing can be made more accurate, thereby improving the accuracy of lip reading recognition.

[0036] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0038] Figure 1A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of another implementation flow of an image processing method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of facial key points provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of another implementation flow of an image processing method provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of another implementation flow of an image processing method provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of an implementation flow of a method for generating a lip reading recognition model provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the structure of a lip reading recognition model provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0046] Figure 9 A schematic diagram of the structure of a device for generating a lip reading recognition model provided in an embodiment of the present application;

[0047] Figure 10 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing this application only and are not intended to limit this application.

[0052] In scenarios where the ambient noise is too loud or it is inconvenient to speak, lip reading recognition can make up for the limitations of speech recognition, thereby enhancing the robustness of human-computer interaction. In the image processing process of lip reading recognition in related technologies, face detection is used to find the corresponding position of the face in the image, and the mouth area in the image is cut out to obtain an image sequence of the mouth area image. This image sequence is input into a three-dimensional convolutional neural network (3D convolutional neural network) for feature extraction, and the features are input into a time series prediction network for classification. However, the image sequence of the mouth area image is not sensitive to mouth movement information, which makes the accuracy of lip reading recognition low. In addition, three-dimensional convolution requires a lot of computing resources and has high hardware requirements, making it difficult to apply on a large scale.

[0053] The present application provides an image processing method that can be executed by a processor of a computer device. The computer device may include a vehicle computer, server, laptop computer, tablet computer, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other device with data processing capabilities. Figure 1 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes the following steps S101 to S104:

[0054] Step S101: Acquire a sequence of image frames containing a mouth object.

[0055] A computer device captures multiple image frames. These frames can be captured by a camera or other capture component while a subject is speaking. The frames are sorted according to the time sequence corresponding to each frame to obtain a raw image frame sequence. The multiple image frames in the image frame sequence contain at least the mouth of the same subject. The subject is typically a human, but can also be another expressive animal, such as an orangutan. In some embodiments, the image frame sequence covers at least the entire process of the subject speaking a sentence. For example, the multiple image frames in the image frame sequence cover at least the process of the subject saying "turn on the music." Furthermore, the number of image frames included in the image frame sequence can be variable, for example, 40, 50, or 100 frames. The raw image frame sequence can be used directly as the image frame sequence for subsequent use; or the raw image sequence can be further processed to obtain a subsequent image frame sequence, for example, by interpolating the raw image sequence to obtain a set number of image frame sequences. Therefore, the image frames in the image frame sequence in various embodiments of the present application can be either real-world captured images or generated based on real-world captured images.

[0056] In some embodiments, the computer device may acquire multiple image frames by calling a camera, or by acquiring the images from other computer devices. For example, if the computer device is a vehicle, images may be acquired through an onboard camera, or images captured by a mobile terminal may be acquired through wireless transmission with the mobile terminal. It should be noted that at least one image frame in the image frame sequence may be derived from a video, and a video may include multiple video frames, each of which corresponds to an image frame. The image frames in the image frame sequence may be continuous image frames, or may be image frames selected from the video at fixed or irregular time intervals. During implementation, multiple pre-captured image frames may be acquired, or multiple image frames may be acquired by performing image capture on a set object in real time. This is not limited here.

[0057] In this way, a sequence of image frames that records the changes in the mouth object when the set subject speaks can be obtained.

[0058] Step S102 : extracting mouth key point features from each image frame in the image frame sequence to obtain mouth key point features of each image frame.

[0059] A mouth key feature extraction process is performed on at least one image frame in the image frame sequence. Position information of the mouth key point associated with the mouth object is extracted from the facial key points in the image frame. Based on the position information of the mouth key point in the at least one image frame, a mouth key point feature corresponding to each image frame is determined, thereby obtaining at least one mouth key point feature in the image frame sequence. The mouth key point feature is calculated based on the position information of the mouth key point, and the position information of the mouth key point is associated with the mouth shape of the mouth object contained in the image. The position information of the same mouth key point in different image frames is associated with the mouth shape of the mouth object in the image frame.

[0060] In some embodiments, the mouth keypoint features corresponding to an image can be determined based on the position information of the mouth keypoints in an image frame by sorting the position information of each mouth keypoint in an image frame according to the corresponding keypoint sequence number to obtain a position sequence, and using the position sequence as the mouth keypoint features. For example, each image frame includes four mouth keypoints, and the coordinates of the mouth keypoints are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively. The mouth keypoint features corresponding to the image frame are determined to be [(x1, y1), (x2, y2), (x3, y3), (x4, y4)].

[0061] In some embodiments, when an image frame sequence includes two image frames, or more than two image frames, a method for determining the mouth key point features corresponding to an image based on the position information of the mouth key points of the image frames may be by calculating the difference information between the position information of the mouth key points of each image frame and the adjacent frames of the image frame, sorting the difference information of each mouth key point in an image frame according to the corresponding key point serial number, and using the sorted sequence as the mouth key point features; wherein the adjacent frames may be the previous image frame and / or the next image frame of the image frame in the image frame sequence, that is, the difference information of the position information includes at least one of the following: difference information between the image frame and the previous image frame; difference information between the image frame and the next image frame. For example, the mouth key point features are determined based on the difference information between the image frame and the previous image frame. Each image frame includes 4 mouth key points. The coordinates of the mouth key points in the first image frame are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and the coordinates of the mouth key points in the second image frame are (x'1, y'1), (x'2, y'2), (x'3, y'3), and (x'4, y'4). In this way, the mouth key point features corresponding to the second image frame are determined to be [(x'1-x1, y'1-y1), (x'2-x2, y'2-y2), (x'3-x3, y'3-y3), (x'4-x4, y'4-y4)].

[0062] Compared to lip reading recognition using sequences of mouth region images, using mouth key point features for lip reading recognition can reduce the computational effort required for image processing, thereby lowering the hardware requirements for the computer executing the image processing method and making the image processing method universally applicable to a wide range of computer devices. Furthermore, using mouth key point features for lip reading recognition can achieve good recognition results for facial images with varying facial shapes, textures, and other appearance characteristics, thereby improving the generalization and accuracy of lip reading recognition.

[0063] Step S103 : generating syllable classification features based on the mouth key point features of the plurality of image frames in the image frame sequence.

[0064] Feature extraction is performed on the key point features of the mouth of multiple image frames in the image frame sequence to obtain a syllable classification feature. The syllable classification feature represents at least one preset syllable category corresponding to the image frame sequence. Each preset syllable category represents at least one syllable with the same or similar mouth shape. In other words, the syllable classification feature can represent the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence. Each element in the syllable classification feature can be used to indicate whether a syllable type exists in the image frame sequence, thereby determining at least one syllable corresponding to the mouth shape contained in the image in the image frame sequence. Here, the syllable types can be pre-divided into a set number of preset syllable categories based on the similarity of the mouth shapes. Each preset syllable category includes at least one syllable type with the same or similar mouth shape. The set number can be set according to the type of language. The similarity of the mouth shape can be determined manually based on experience or through machine learning. Taking Chinese as an example, without considering tones, there are 419 syllable types in Chinese characters. These 419 syllables can be divided into 100 categories based on the corresponding mouth shapes, and the length of the corresponding syllable classification feature is 100. For other languages, such as English, the syllable types can be divided into a set number of preset syllable categories based on phonetic symbols, and the length of the syllable classification feature can be set according to the correspondence between syllables and mouth shapes.

[0065] In some embodiments, spatiotemporal features can be extracted from at least two mouth key point features in an image frame sequence to obtain spatiotemporal features corresponding to each mouth key point feature, and syllable classification features can be determined based on the spatiotemporal features. Here, spatiotemporal features can be extracted using a time series prediction network and / or a fully convolutional network to obtain spatiotemporal features corresponding to each mouth key point feature. In some possible implementations, at least two spatiotemporal features can be concatenated using a flattening layer or other method, and the concatenated spatiotemporal features can then be classified to obtain syllable classification features.

[0066] In this way, syllable classification features are extracted from the key point features of the mouth. The syllable classification features can represent at least one syllable corresponding to the mouth shape of the mouth object in the image frame sequence. Using the syllable classification features to assist lip reading recognition can improve the accuracy of lip reading recognition.

[0067] Step S104: determining keywords matching the syllable classification features in a preset keyword library.

[0068] In some embodiments, a certain number of keywords are pre-set in the keyword library for matching with the syllable classification features to obtain the image processing results of lip reading recognition. The keyword can be determined by directly outputting the keyword or by outputting the keyword's sequence number in the keyword library.

[0069] In some possible implementations, the preset keyword library can be set according to specific application scenarios. In a driving scenario, the keywords can be set to "turn on the audio", "open the left window", etc. It should be noted that the preset keyword library represents the storage format of the keywords.

[0070] In some possible implementations, the detection results obtained by speech detection and the recognition results obtained by lip reading recognition can be combined to determine matching keywords; for example, the weights of the detection results of speech detection and the recognition results of lip reading recognition are set separately, and the weighted calculation results are used as the matching basis.

[0071] In this way, by representing the syllable classification features corresponding to the image frame sequence, the keywords of the words corresponding to the syllables are determined according to the syllable categories represented by the syllable classification features, thereby improving the accuracy of the keywords obtained by image processing.

[0072] In an embodiment of the present application, mouth key point features are extracted from the mouth key point features of the image frames in the image frame sequence, and syllable classification features corresponding to the image frame sequence are generated using the mouth key point features. Keywords are then matched in a preset keyword library based on the syllable classification features. In this way, the amount of computation required for the image processing process of lip reading recognition can be reduced, thereby reducing the hardware requirements for computer equipment. At the same time, good recognition results can be achieved for facial images with different appearance information such as face shapes and textures, thereby improving the generalization ability of lip reading recognition. In addition, by representing the syllable classification features corresponding to the image frame sequence, the keywords corresponding to the syllables are determined based on the syllable categories represented by the syllable classification features. This can make the keywords obtained by image processing more accurate, thereby improving the accuracy of lip reading recognition.

[0073] In some possible implementations, the speaking interval of a set subject in a video is detected by lip movement recognition processing to obtain an image frame sequence covering the speaking process of the set subject. That is, the above step S101 can be implemented by the following steps S1011 and S1012:

[0074] Step S1011: Acquire a video whose image frame includes the mouth object.

[0075] The computer device shoots the set object through a camera or other acquisition components to obtain a video of the mouth object.

[0076] Step S1012: performing lip movement recognition on the mouth object, and determining a plurality of video frames in which the mouth object is in a speaking state as an image frame sequence.

[0077] The video is cropped using lip movement recognition technology to obtain a video recording the speaking process of the set object. The image of the video frame includes the mouth object in the speaking state, and multiple video frame images are selected from the cropped video as an image frame sequence.

[0078] In the above scheme, the image frame sequence can at least cover the complete process of the set subject speaking, and through lip movement recognition, the image frames irrelevant to the speaking process included in the image frame sequence are reduced. The image frame sequence obtained in this way is used for image processing to obtain matching keywords, which can further improve the accuracy of lip reading recognition and reduce the amount of calculation required for the image processing process of lip reading recognition.

[0079] As mentioned above, the number of image frames included in the image frame sequence of image processing may not be fixed. In some possible implementations, the original image sequence obtained by acquisition may be interpolated to obtain an image frame sequence including a preset number of image frames.

[0080] In some implementations, the following steps S1013 or S1014 may be included:

[0081] Step S1013 , performing image interpolation on the acquired original image sequence containing the mouth object to obtain the image frame sequence.

[0082] The method of performing interpolation processing on the original image sequence obtained by the acquisition to obtain an image frame sequence including a preset number of image frames can be to perform image interpolation processing based on the image frames in the original image sequence to generate a preset number of image frames, and obtain an image frame sequence for subsequent mouth key point feature extraction based on the generated image frames and / or the acquired image frames.

[0083] Step S1014 : Based on the key points of the mouth in the acquired original image sequence containing the mouth object, interpolate the original image sequence to obtain the image frame sequence.

[0084] The method of performing interpolation processing on the original image sequence obtained by the acquisition to obtain an image frame sequence including a preset number of image frames can be to generate newly inserted image frames based on the position information of the mouth key points in the original image sequence, and the position information of the mouth key points in the newly inserted image frames is predicted based on the position information of the mouth key points in the original image sequence, thereby realizing interpolation of the original image sequence, obtaining the preset number of key point information corresponding to the image frame sequence, and realizing subsequent mouth key point feature extraction.

[0085] Among them, the number of image frames can be pre-set based on experience. The larger the preset number, the higher the recognition accuracy, but the more computing resources are consumed, which affects the hardware operation efficiency. Taking into account the accuracy, hardware operation efficiency and the number of keywords, the preset number can be set to 60 in actual applications.

[0086] In this way, lip reading recognition is performed using an interpolated image frame sequence, and no requirement is placed on the number of frames of the original image sequence acquired during the lip reading recognition process, thereby improving the robustness of the image recognition method during the lip reading recognition process.

[0087] In some possible implementations, the position information of the mouth key points in each image frame and the adjacent frames is used to determine the mouth key point features of the image frame, that is, the above step S102 can be performed by Figure 2 The steps shown are implemented. Figure 2 This is another implementation flow diagram of the image processing method provided in the embodiment of the present application, combined with Figure 2 The steps shown are explained below:

[0088] Step S201 : determining position information of at least two mouth key points of the mouth object in each image frame.

[0089] The image frame sequence includes at least two image frames, and the position information of the mouth key points associated with the mouth object in each image frame is extracted. The number of mouth key points is at least two, and they are distributed at least on the upper and lower lips in the image. The specific setting is usually related to the key point recognition algorithm. For example, the number of mouth key points for 68-point key point detection is 16. The position information of each mouth key point can be represented by a position parameter, for example, by a two-dimensional coordinate in an image coordinate system, and the two-dimensional coordinate includes width (horizontal coordinate) and height (vertical coordinate). Here, the position information of the mouth key point is related to the mouth shape of the mouth object contained in the image, and the position information of the same mouth key point in different images changes with the change of the mouth shape. Figure 3For example, the 106 facial keypoints shown in the diagram below include keypoints 0-105, which describe facial features such as the facial contour, eyebrows, eyes, nose, and mouth. Keypoints 84-103 are used to describe the mouth. Keypoint 93 is located at different positions in the two frames corresponding to different speech content. When keypoint 93 has a smaller vertical coordinate in the image, it indicates a wider mouth opening and is more likely to correspond to "ah" rather than "oh."

[0090] Step S202 : for each image frame in the image frame sequence, determine the mouth key point features corresponding to the image frame according to the position information of the mouth key points in the image frame and its adjacent frames.

[0091] For each first image frame in the image frame sequence, the mouth key point features corresponding to each mouth key point in the first image frame can be calculated using the position information of the mouth key points in at least two image frames including the first image frame. The mouth key point features can include inter-frame difference information and / or intra-frame difference information. The first image frame can be any image frame in the image frame sequence. The inter-frame difference information can represent the difference information between the position information of the same mouth key point in different image frames, and the intra-frame difference information can represent the difference information between the position information of different mouth key points in the same image frame. Here, the inter-frame difference information of each mouth key point in the first image frame and the position information of the mouth key point in adjacent frames of the first image frame are calculated; and / or the intra-frame difference information of the mouth key point in the first image frame is calculated using the position information of at least two mouth key points in the first image frame.

[0092] Compared with using a mouth area image sequence for lip reading recognition, the embodiment of the present application uses the position information of multiple mouth key points in multiple image frames to obtain mouth key point features, so that the mouth key point features can represent the change process of the mouth key points during the speaking process corresponding to the image frame sequence, thereby better extracting the change features of the mouth shape during the speaking process; in this way, using mouth key point features for lip reading recognition can improve the accuracy of lip reading recognition.

[0093] In some possible implementations, the mouth key point features are determined by utilizing the differences in position information of the mouth key points in adjacent frames and the differences in position information of a preset pair of mouth key points in the same image frame. That is, the above step S202 can be implemented by the following steps S2021 and S2022:

[0094] Step S2021: For each of the mouth key points, based on the position information of the mouth key point in the image frame and the position information of the mouth key point in the adjacent image frame of the image frame, determine the first height difference and / or first width difference of the mouth key point between the image frame and the adjacent frame as the inter-frame difference information of the mouth key point.

[0095] In some embodiments, when calculating the mouth key point features corresponding to each first image frame, for each mouth key point, based on the position information of the mouth key point in the first image frame and the position information of each second image frame of at least one second image frame, the difference information between the position information of the mouth key point in the first image frame and the two second image frames is calculated. The second image frame is an image frame adjacent to the first image frame, that is, an adjacent frame of the first image frame; the difference information can be a first height difference, a first width difference, or a combination of the first height difference and the first width difference; the first width difference is the width difference of the mouth key point between the two image frames (the first image frame and the second image frame), and the first height difference is the height difference of the mouth key point between the two image frames. In some possible implementations, when calculating the difference, it can be set as the position information of the subsequent image frame minus the position information of the previous image frame, or it can be set as the position information of the previous image frame minus the position information of the subsequent image frame. Then, for each mouth key point, using the first image frame and each second image frame in at least one second image frame, the same amount of difference information as the second image frames can be obtained, and these difference information are determined as the inter-frame difference information of this mouth key point in the first image frame.

[0096] For example, the coordinates of a mouth key point in three consecutive image frames are (x1, y1), (x'1, y'1), and (x"1, y"1). The second image frame is taken as the first image frame, and the first and third image frames before and after are taken as the second image frames. The first height difference and the first width difference are calculated, and the inter-frame difference information of the mouth key point in the first image frame is obtained as (x'1-x1, y'1-y1, x"1-x'1, y"1-y'1).

[0097] Step S2022 : For each mouth key point, determine intra-frame difference information of the mouth key point based on a second height difference and / or a second width difference between the mouth key point and other mouth key points of the same mouth object in the image frame.

[0098] In some embodiments, when determining the mouth key point features corresponding to each first image frame, for each mouth key point, a second height difference and / or a second width difference between the mouth key point and other mouth key points of the same mouth object is calculated, and the second height difference and / or the second width difference is determined as the intra-frame difference information of each mouth key point in the corresponding preset mouth key point pair in the first image frame. The other mouth key points may be fixed mouth key points, such as the mouth key points corresponding to the lip bead, for example. Figure 3 The key point 98 shown can also be a mouth key point that satisfies a set positional relationship with each mouth key point. Two mouth key points constitute a preset mouth key point pair. Furthermore, when setting the preset mouth key points, the positional information of the mouth key points in the image can be considered. That is, the two mouth key points belonging to the same preset mouth key point pair must satisfy the set positional relationship. For example, two mouth key points located on the upper and lower lips of a mouth object can be determined as a mouth key point pair. Furthermore, two mouth key points whose image width difference information is less than a preset value can also be determined as a preset mouth key point pair. In this way, the second height difference of the preset mouth key point pair can better represent the mouth shape of the mouth object in the first image frame.

[0099] In some possible implementations, a mouth key point can form a preset mouth key point pair with two or more mouth key points. That is, each mouth key point can belong to more than one mouth key point pair. In this case, the second height difference of each mouth key point pair to which the mouth key point belongs is determined respectively, and the intra-frame difference information of the mouth key point in the first image frame is determined by weighted summation of at least two second height differences. Figure 3 Taking the schematic diagram of 106 facial key points as an example, key point 86 can form a preset mouth key point pair with key points 103 and 94. In other words, key point 86 belongs to both mouth key point pairs. The second height difference of each mouth key point pair is calculated separately, and then the two second height differences are weighted summed to determine the intra-frame difference information of key point 86 in the first image frame. This can improve the calculation deviation of mouth key point features caused by key point recognition errors, and lip reading recognition based on such mouth key point features can improve lip reading recognition accuracy.

[0100] In some possible implementations, inter-frame difference information and intra-frame difference information of a mouth key point in the first image frame are respectively obtained through steps S2021 and S2022. The inter-frame difference information and the intra-frame difference information can be spliced ​​to obtain an element of the mouth key point feature corresponding to the mouth key point in the first image frame, thereby determining the mouth key point feature corresponding to the first image frame based on the inter-frame difference information and intra-frame difference information of all mouth key points in the first image frame.

[0101] In an embodiment of the present application, the mouth key point features are obtained by utilizing the inter-frame difference information of the position information of each mouth key point in adjacent image frames and the intra-frame difference information of the position information of this mouth key point and the preset mouth key point, so that the mouth key point features can represent the differences between the mouth key points that satisfy the set relationship, thereby improving the accuracy of determining the lip shape in each frame of the image; moreover, the mouth key point features can also represent the inter-frame change process of the mouth key points during the speaking process corresponding to the image frame sequence; in this way, the change characteristics of the lip shape during the speaking process can be better extracted, thereby improving the accuracy of lip reading recognition.

[0102] In some possible implementations, spatiotemporal features are extracted based on the mouth key point features of the image frame sequence to obtain the spatiotemporal features corresponding to the mouth object in each image frame, and syllable features are classified based on the spatiotemporal features to obtain the syllable classification features corresponding to the mouth object. That is, the above step S103 can be performed by Figure 4 The steps shown are implemented. Figure 4 This is another implementation flow diagram of the image processing method provided in the embodiment of the present application, combined with Figure 4 The steps shown are explained below:

[0103] Step S401 : performing spatial feature extraction on the mouth key point features of each image frame to obtain the spatial features of the mouth object in each image frame.

[0104] As mentioned above, at least one mouth keypoint feature can be obtained from a sequence of image frames. Each mouth keypoint feature is calculated based on the positional information of the mouth keypoint, which represents the position of the mouth object within an image frame. Each mouth keypoint feature corresponds to a specific image frame. For each mouth keypoint feature, the spatial features of the mouth object in the corresponding image frame can be extracted from the mouth keypoint feature using any suitable feature extraction method. For example, convolutional neural networks, recurrent neural networks, and other methods can be used to extract the spatial features.

[0105] In some possible implementations, the speaking interval of a set subject in a video is detected by lip movement recognition processing to obtain an image frame sequence covering the speaking process of the set subject. That is, the above step S401 can be implemented by the following steps S4011 and S4012:

[0106] Step S4011 , fusing the inter-frame difference information and the intra-frame difference information of the multiple mouth key points of the mouth object to obtain the inter-frame difference features and the intra-frame difference features of the mouth object in each image frame.

[0107] As mentioned above, each mouth keypoint feature is calculated from the position information of the mouth keypoint. The position information of the mouth keypoint represents the position of the mouth object in an image frame, and each mouth keypoint feature corresponds to one image frame. Inter-frame difference information can represent the difference in position information of the same mouth keypoint in different frames, and intra-frame difference information can represent the difference in position information between different mouth keypoints in the same frame. In some embodiments, the inter-frame difference information of multiple mouth keypoints in each image frame is fused, and the intra-frame difference information of multiple mouth keypoints in each image frame is fused to obtain the inter-frame difference features and intra-frame difference features of the mouth object in each image frame. The inter-frame difference information and / or intra-frame difference information can be fused using a convolutional neural network, a recurrent neural network, or the like, using a convolution kernel of a preset size to fuse the information of multiple mouth keypoints, thereby achieving the fusion of the inter-frame and / or frame difference information of multiple mouth keypoints.

[0108] For example, a mouth key point corresponds to an element in the mouth key point feature, including a 5-dimensional feature. The first 4 dimensions of the 5-dimensional feature are inter-frame difference information, namely the width difference between the first image frame and the previous image frame, the height difference between the first image frame and the previous image frame, the width difference between the first image frame and the next image frame, and the height difference between the first image frame and the next image frame. The fifth dimension is the intra-frame difference information. Feature extraction is performed on each dimension of the 5-dimensional feature at least two mouth key points (that is, between the elements of the mouth key point feature). The first 4 dimensions of the obtained features are the inter-frame difference features of the mouth object in this image frame, and the fifth dimension is the intra-frame difference feature of the mouth object in this image frame.

[0109] Step S4012: fusing the inter-frame difference features and intra-frame difference features of the mouth object in the plurality of image frames to obtain the spatial features of the mouth object in each image frame.

[0110] In some embodiments, the method for fusing the inter-frame difference features and intra-frame difference features of multiple image frames can be a convolutional neural network, a recurrent neural network, etc., which uses a convolution kernel of a preset size to fuse the information of multiple mouth key points to achieve the fusion of the inter-frame difference information and the intra-frame difference information of each mouth key point, and obtain the spatial features of the mouth object in this image frame.

[0111] In the above steps S4011 to S4012, the inter-frame difference information and the intra-frame difference information of at least two mouth key points of the mouth object in each image frame are respectively fused to obtain the inter-frame difference features representing the inter-frame difference information between the mouth key points, and the intra-frame difference features representing the intra-frame difference information between the mouth key points. The inter-frame difference features and the intra-frame difference features of the mouth key points in each image frame are then feature fused, which can better extract the spatial features of the mouth object in each image frame, thereby improving the accuracy of determining the mouth shape in each frame image.

[0112] Step S402 : performing temporal feature extraction on the spatial features of the mouth object in the plurality of image frames to obtain the spatiotemporal features of the mouth object.

[0113] In some embodiments, for each third image frame in the at least one image frame, feature extraction can be performed using the spatial features of the mouth object in at least two image frames, including the third image frame, to obtain the spatiotemporal features corresponding to the mouth object in the third image frame. The spatiotemporal features of the mouth object can be extracted from the spatial features using any suitable feature extraction method. For example, temporal features can be extracted using a convolutional neural network, a recurrent neural network, or the like to obtain the spatiotemporal features.

[0114] In some possible implementations, temporal feature extraction of the spatial features of the mouth object in multiple image frames can be performed multiple times. Taking one temporal feature extraction as an example, feature extraction is performed using a 1×5 convolution kernel. Each convolution is performed to extract the spatial features of two image frames before and after the third image frame. The extracted spatiotemporal features include information of five image frames.

[0115] As the number of times the temporal features are extracted increases and the convolution kernel used becomes larger, the spatiotemporal features corresponding to each image frame can represent more information of the image frames, allowing information between frames to be exchanged. As a result, the corresponding receptive field becomes larger, which is conducive to learning the words composed of multiple frames of images and the time sequence between different words, and can improve the accuracy of lip reading recognition. However, it requires more computing resources, which affects the hardware computing efficiency. Taking into account both accuracy and hardware computing efficiency, the number of image feature extractions can be set to 5 in actual applications.

[0116] Step S403 : extracting syllable classification features based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

[0117] In some embodiments, syllable classification features are extracted from the spatiotemporal features corresponding to each image frame of the mouth object in at least two image frames to obtain syllable classification features of the mouth object; wherein the syllable classification feature can represent at least one syllable corresponding to the mouth shape of the mouth object during speaking, and each element in the syllable classification feature is used to indicate whether a preset syllable type exists during speaking, thereby determining at least one syllable corresponding to the mouth shape contained in the image in the image frame sequence. The syllable classification features of the mouth object can be extracted from the spatiotemporal features using any suitable feature extraction method. For example, a fully connected layer, a global average pooling layer, etc. can be used to extract syllable classification features from the spatiotemporal features to obtain syllable classification features.

[0118] The embodiments of the present application support the use of convolutional neural networks for spatiotemporal feature extraction. Compared to extracting spatiotemporal features using time series prediction networks such as recurrent neural networks (RNNs), the computational complexity of extracting spatiotemporal features using convolutional neural networks is lower, which can reduce the consumption of computing resources and lower the hardware requirements of computer devices for lip reading recognition. In particular, the use of convolutional neural networks can reduce the requirements for chip computing power, thereby enabling the image processing methods provided in the embodiments of the present application to be implemented using more lightweight chips, enabling more hardware to support the image processing methods used in the lip reading recognition process of the embodiments of the present application, thereby improving the versatility of lip reading recognition. For example, computer devices such as in-vehicle computers can also implement lip reading recognition.

[0119] The present application embodiment provides an image processing method, which can be executed by a processor of a computer device. Figure 5 As shown, the method includes the following steps S501 to S504:

[0120] Step S501: Acquire a sequence of image frames containing a mouth object.

[0121] Here, step S501 corresponds to the aforementioned step S101, and the specific implementation of the aforementioned step S101 may be referred to during implementation.

[0122] Step S502 : extracting mouth key point features from each image frame in the image frame sequence to obtain mouth key point features of each image frame.

[0123] Here, step S502 corresponds to the aforementioned step S102, and the specific implementation of the aforementioned step S102 may be referred to during implementation.

[0124] Step S503 : Using a trained syllable feature extraction network, mouth key point features of a plurality of image frames in the image frame sequence are processed to obtain syllable classification features.

[0125] During implementation, the syllable feature extraction network can be any suitable network for feature extraction, including but not limited to convolutional neural networks, recurrent neural networks, etc.; those skilled in the art can select a suitable network structure for the syllable feature extraction network based on actual conditions, and the embodiments of this application are not limited thereto.

[0126] Step S504: using the trained classification network, determine keywords that match the syllable classification features in the preset keyword library.

[0127] During implementation, the classification network can be any suitable network for feature classification, such as a global average pooling layer or a fully connected layer. Those skilled in the art can select a suitable network structure for the classification network according to actual conditions, and the embodiments of the present application are not limited thereto.

[0128] In the embodiments of the present application, a trained syllable feature extraction network is used to process mouth key point features to obtain syllable classification features. This trained classification network is then used to identify keywords matching the syllable classification features from a preset keyword library. Because each network in the deep learning model is learnable, the accuracy of the extracted syllable classification features and the keywords matching them can be improved, thereby making the keywords obtained from image processing more precise and improving the accuracy of lip reading recognition.

[0129] In some possible implementations, the syllable feature extraction network includes a spatial feature extraction subnetwork, a temporal feature extraction subnetwork, and a classification feature extraction subnetwork. That is, the above step S503 can be implemented by the following steps S5031 to S5033:

[0130] Step S5031: Using the spatial feature extraction sub-network, spatial feature extraction is performed on the mouth key point features of each image frame to obtain the spatial features of the mouth object in each image frame.

[0131] During implementation, the spatial feature extraction subnetwork can be any suitable network for image feature extraction, including but not limited to convolutional neural networks, recurrent neural networks, etc. Those skilled in the art can select an appropriate network structure based on the actual method of performing spatial feature extraction on each of the mouth key point features, and this embodiment of the present application is not limiting.

[0132] Step S5032: Using the temporal feature extraction sub-network, temporal feature extraction is performed on the spatial features of the mouth object in the plurality of image frames to obtain the spatiotemporal features of the mouth object.

[0133] Here, the temporal feature extraction subnetwork can be any suitable network for image feature extraction, including but not limited to a convolutional neural network, a recurrent neural network, etc. Those skilled in the art can select a suitable network structure based on the actual method of performing at least one temporal feature extraction on the spatial features of the mouth object in at least one image frame, and the embodiments of the present application are not limited thereto.

[0134] Step S5033: Using the classification feature extraction subnetwork, syllable classification features are extracted based on the spatiotemporal features of the mouth object to obtain the syllable classification features of the mouth object.

[0135] Here, the classification feature extraction subnetwork can be any suitable network for feature classification, such as a global average pooling layer or a fully connected layer. Those skilled in the art can select an appropriate network structure based on the actual method of extracting classification features for each spatiotemporal feature of the mouth object, and the embodiments of this application are not limited thereto.

[0136] The present application embodiment provides a method for generating a lip reading recognition model, which can be executed by a processor of a computer device. Figure 6 As shown, the method includes the following steps S601 to S604:

[0137] Step S601: Acquire a sample image frame sequence containing a mouth object.

[0138] In some embodiments, a computer device obtains a sequence of sample image frames labeled with a keyword. The sequence of sample image frames includes multiple sample image frames, and the sample images in the sequence of sample image frames are sorted according to the time sequence corresponding to each sample image frame. Furthermore, the number of sample image frames included in the sequence of sample image frames may not be fixed. For example, the number of sample image frames in the sequence of sample image frames may be 40, 50, or 100.

[0139] In this way, a sample image frame sequence that at least covers the entire process of the set subject speaking a sentence can be obtained.

[0140] Step S602 : extracting mouth key point features from each sample image frame in the sample image frame sequence to obtain mouth key point features of each sample image frame.

[0141] Mouth key point extraction is performed on at least one sample image frame in the sample image frame sequence, and position information of the mouth key point associated with the mouth object is extracted from the facial key points of the sample image frame. Based on the position information of the mouth key point of the at least one sample image frame, a mouth key point feature corresponding to each sample image frame is determined, thereby obtaining at least one mouth key point feature for the sample image frame sequence. The mouth key point feature is calculated based on the position information of the mouth key point, and the position information of the mouth key point is associated with the mouth shape of the mouth object contained in the sample image. The position information of the same mouth key point in different sample image frames is related to the mouth shape of the mouth object in the sample image frame.

[0142] In some embodiments, a method for determining the mouth key point features corresponding to the sample image based on the position information of the mouth key points of the sample image frame can be to sort the position information of each mouth key point in a sample image frame according to the corresponding key point serial number to obtain a position sequence, and use the position sequence as the mouth key point feature.

[0143] In some embodiments, when the sample image frame sequence includes two sample image frames, or more than two sample image frames, the method of determining the mouth key point features corresponding to the sample image based on the position information of the mouth key points of the sample image frames can be by calculating the difference information of the position information of the mouth key points of each sample image frame and the adjacent frames of the sample image frame, sorting the difference information of each mouth key point in a sample image frame according to the corresponding key point sequence number, and using the sorted sequence as the mouth key point feature; wherein the adjacent frames can be the previous sample image frame and / or the next sample image frame of the sample image frame.

[0144] Here, steps S601 to S602 correspond to the aforementioned steps S101 to S102 respectively, and the specific implementations of the aforementioned steps S101 to S102 may be referred to during implementation.

[0145] Step S603 , using the model to be trained, generating syllable classification features according to the mouth key point features of multiple sample image frames in the sample image frame sequence, and determining keywords matching the syllable classification features in a preset keyword library.

[0146] The syllable classification feature represents the syllable category corresponding to the mouth shape of the mouth object in the sample image frame sequence.

[0147] Here, the model to be trained can be any suitable deep learning model, which is not limited here. During implementation, those skilled in the art can use a suitable network structure to construct the model to be trained according to actual conditions.

[0148] The process of using the model to be trained to process the mouth key point features of multiple sample image frames in the sample image frame sequence to generate syllable classification features, the syllable classification features characterize the syllable category corresponding to the mouth shape of the mouth object in the sample image frame sequence, and determining the keywords matching the syllable classification features in the preset keyword library corresponds to the process of processing the mouth key point features in steps S103 to S104 in the aforementioned embodiment. During implementation, the specific implementation methods of the aforementioned steps S103 to S104 can be referred to.

[0149] In this way, syllable-assisted learning can effectively reduce the learning difficulty of keyword recognition and classification, thereby improving the accuracy of lip reading recognition.

[0150] Step S604: Based on the determined keywords and keyword tags, the network parameters of the model are updated at least once to obtain a trained lip reading recognition model.

[0151] Here, based on the determined keywords and keyword labels, it can be determined whether to update the network parameters of the model. If it is determined that the network parameters of the model are to be updated, an appropriate parameter learning difficulty update algorithm is used to update the network parameters of the model, and the matching keywords are re-determined using the model after the parameter update. Based on the re-determined keywords and keyword labels, it is determined whether to continue updating the network parameters of the model. If it is determined that the network parameters of the model are not to be further updated, the final updated model is determined as the trained lip reading recognition model.

[0152] In some embodiments, a loss value can be determined based on the determined keywords and keyword tags, and if the loss value does not meet the preset conditions, the network parameters of the model are updated. If the loss value meets the preset conditions or the number of updates to the network parameters of the model reaches a set threshold, the updating of the network parameters of the model is stopped, and the final updated model is determined as the trained lip reading recognition model. The preset conditions may include, but are not limited to, at least one of the loss value being less than a set loss threshold, the change in the loss value converging, etc. During implementation, the preset conditions can be set according to actual conditions, and the embodiments of the present application are not limited to this.

[0153] The method for updating the network parameters of the model can be determined according to actual conditions, and may include but is not limited to at least one of the gradient descent method, Newton momentum method, etc., which is not limited here.

[0154] In embodiments of the present application, during model training, syllable-assisted learning can effectively reduce the learning difficulty of keyword recognition and classification, thereby improving the accuracy of lip reading recognition by a trained lip reading recognition model. Furthermore, because syllable classification features are determined based on mouth key point features, they can better reflect the syllables corresponding to the mouth shapes in the image frame sequence. Using syllable classification features to assist lip reading recognition makes the keywords obtained through image processing more accurate, thereby improving the accuracy of lip reading recognition. Furthermore, compared to lip reading recognition using a mouth region image sequence obtained by cropping facial images, using mouth key point features for lip reading recognition can reduce the computational effort required during image processing, thereby lowering the hardware requirements for the computer device executing the image processing method. Furthermore, good recognition results can be achieved for facial images with different facial shapes, textures, and other appearance information. Therefore, based on mouth key point features, the recognition capability of image categories with facial shapes and textures not considered during model training can be improved, thereby improving the generalization capability of lip reading recognition.

[0155] In some embodiments, the model includes a syllable feature extraction network and a classification network, and the above step S603 may include the following steps S6031 to S6032:

[0156] Step S6031 : Using the syllable feature extraction network, a syllable classification feature is generated based on the mouth key point features of a plurality of sample image frames in the sample image frame sequence.

[0157] Step S6032: Using the classification network, determine keywords that match the syllable classification features in a preset keyword library.

[0158] Here, steps S6031 to S6032 correspond to the aforementioned steps S503 to S504 respectively, and when implementing, reference may be made to the specific implementation of the aforementioned steps S503 to S504.

[0159] In some embodiments, the syllable feature extraction network includes a spatial feature extraction subnetwork, a temporal feature extraction subnetwork, and a syllable classification feature extraction subnetwork. Step S6031 may include the following steps S60311 to S60313:

[0160] Step S60311: Utilize the spatial feature extraction sub-network to perform spatial feature extraction on the mouth key point features of each sample image frame to obtain the spatial features of the mouth object in each sample image frame.

[0161] Step S60312: Using the temporal feature extraction sub-network, sample temporal feature extraction is performed on the spatial features of the mouth object in the plurality of sample image frames to obtain the spatiotemporal features of the mouth object.

[0162] Step S60313: Utilize the syllable classification feature extraction subnetwork to extract syllable classification features based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

[0163] Here, steps S60311 to S60313 correspond to the aforementioned steps S5031 to S5033 respectively, and the specific implementation methods of the aforementioned steps S5031 to S5033 may be referred to during implementation.

[0164] The following describes the application of the image processing method provided in the embodiments of the present application in actual scenarios, taking image processing for Chinese lip reading recognition as an example.

[0165] Figure 7 This is a schematic diagram of the structure of a lip reading recognition model provided in an embodiment of the present application. Figure 7 As shown, the lip reading recognition model structure includes: a single-frame feature extraction network 701, an inter-frame feature fusion network 702 and a feature sequence classification network 703. Among them, the single-frame feature extraction network 701 includes a spatial feature extraction network 7011 and a spatial feature fusion network 7012, and the feature sequence classification network 703 includes a syllable feature layer 7031 and a first linear layer 7032. The embodiment of the present application provides an image processing method, which generates an image frame sequence of an object speaking according to the lip movement recognition detection results, uses the features of the key points of the face as the input of the lip reading recognition model, uses single syllables to assist in detecting the syllables in the speaking sequence, and uses the syllable feature layer to realize the classification of the speaking sequence. The following is combined with Figure 7 The image processing method of the embodiment of the present application is described.

[0166] The present application provides an image processing method that can be executed by a processor of a computer device. The computer device may be a device with data processing capabilities, such as a vehicle computer. The image processing method may include the following steps 1 to 4:

[0167] Step 1: Input preprocessing.

[0168] The input video sequence obtained by the computer device is a non-fixed frame, and the video sequence may include a non-fixed number of video frames. The key point sequence corresponds to 106 key points in each image frame. The 20 key points of the mouth object are taken out, and then an interpolation method (for example, bilinear interpolation) is used to generate a position sequence of key points with a length of 60 image frames. The 20 mouth key points are used as feature dimensions, and each key point in the position sequence corresponds to a feature of length 5 in each image frame, thereby obtaining a mouth key point feature 704 corresponding to 60 frames. Each frame of the mouth key point feature 704 corresponds to one image frame, and each of the 20 key points corresponds to a 5-dimensional feature in each image frame.

[0169] Specifically, the first four dimensions of the feature are obtained based on the coordinate difference between this frame and the previous and next frames, and the fifth dimension of the feature is obtained based on the height difference between the preset key point pairs in this frame. The first four dimensions can reflect the changes in lip shape between the current frame and the previous and next frames, while the fifth dimension reflects the lip shape of the current frame. Here, the captured video can be processed using methods such as lip movement recognition so that each video can at least cover the process of a set subject (usually a person) speaking a sentence, and each sentence corresponds to a keyword. In this way, there is a one-to-one relationship between the video and the keyword. Moreover, regardless of the number of frames in the acquired video speech sequence, a 60-frame position sequence can be obtained using interpolation.

[0170] Here, a larger number of position sequence frames reduces computational efficiency but improves lip reading recognition performance. Taking into account performance, computational efficiency, and the word count distribution of the keywords to be detected, the number of position sequence frames is set to 60. The performance can be the accuracy of lip reading recognition.

[0171] Step 2: Single frame feature extraction.

[0172] Computer equipment through Figure 7 The single-frame feature extraction network 701 implements single-frame feature extraction. The single-frame feature extraction network 701 includes a spatial feature extraction network 7011 and a spatial feature fusion network 7012.

[0173] The mouth key point features 704 are input into the lip reading recognition model. Using the spatial feature extraction network 7011, feature extraction is performed independently on each frame of the mouth key point features 704 using a 1×1 convolution kernel. This convolution is repeated twice, and the extracted features are input into the spatial feature fusion network 7012. In the spatial feature fusion network 7012, a 5×1 convolution kernel is first used to fuse the five features of each key point to obtain the spatial features of each image frame. This yields the features 705 extracted by the spatial feature extraction network 7011 for each image frame. A 1×1 convolution kernel is then used to fuse the features between the 20 key points to obtain the spatial features 706 of the image frame, completing the feature extraction of a single frame.

[0174] In some possible implementations, the convolution kernel may be a residual block kernel (Residual Blockkernel).

[0175] Step three: inter-frame feature fusion.

[0176] Computer equipment through Figure 7 The inter-frame feature fusion network 702 realizes inter-frame feature fusion of adjacent image frames.

[0177] The spatial features 706 of each image frame are input into the inter-frame feature fusion network 702, and convolution is performed on the sequence length dimension using a 1×5 convolution kernel to fuse the spatial features 706 of each image frame with the spatial features 706 of the two previous and next image frames. The above convolution is repeated 5 times to improve the receptive field, so that information between frames can be exchanged and the association between adjacent frames is strengthened, which is conducive to learning the time series between keywords and Chinese characters composed of multiple frames.

[0178] This step consumes a certain amount of computing resources. To improve performance, the convolution kernel size can be increased, and the number of repetitions can be increased, which will correspondingly affect efficiency. Considering both accuracy and hardware efficiency, in practice, the number of extractions can be set to 5 and the convolution kernel size to 5.

[0179] Step 4: Feature sequence classification.

[0180] Computer equipment through Figure 7 The feature sequence classification network 703 classifies the feature sequence and obtains the keyword sequence number corresponding to the video sequence. The feature sequence includes the spatiotemporal features of multiple image frames. The feature sequence classification network 703 includes a syllable feature layer 7031 and a first linear layer 7032.

[0181] The spatiotemporal features are input into a "flattening layer + second linear layer + nonlinear activation (ReLU) layer" for processing. The spatiotemporal features of all image frames are fused into a one-dimensional vector 707, achieving feature fusion of the spatiotemporal features. The one-dimensional vector 707 is input into the third linear layer for 100-class monosyllable auxiliary classification to obtain syllable classification features. The syllable classification features are input into the first linear layer 7032, which outputs the keyword number of the video sequence to be detected. The third linear layer can be trained using the softmax function with the binary cross entropy loss (BCE loss) function as the loss function. The first linear layer 7032 can be trained with the focal loss function as the loss function and predicted using the softmax function. In practical applications, the first linear layer 7032 can be a margin linear layer implemented by a fully connected layer or a global average pooling layer. Compared to using a global average pooling layer, directly expanding the fully connected layer is equivalent to corresponding to a learnable position embedding for each frame, thereby recording the position and sequence information of each frame in the sentence.

[0182] In some possible implementations, a detection algorithm for lip reading recognition using syllable-assisted learning is used. At present, there are 419 categories of pronunciation for all Chinese characters without considering the tone. These 419 categories of syllables are divided into 100 categories according to the mouth shape. Syllables with the same mouth shape are classified into the same category. A feature with a length of 100 (corresponding to the syllable classification feature in the aforementioned embodiment) is placed before the fully connected layer of the last classification, and the output of the feature is used as auxiliary supervision for the 100 classifications. At this time, the output of the syllable feature layer 7031 represents which syllables are in the lip reading sequence. Classifying the output results of the syllable feature layer 7031 can effectively reduce the learning difficulty of the fully connected layer classification, thereby improving performance. Among them, the syllable feature layer 7031 can be implemented using a linear layer.

[0183] In the embodiment of the present application, the monosyllabic auxiliary strategy significantly improves performance; and these keywords used for matching can be stored in the form of a preset keyword library, which is convenient for adding new keywords for matching later. When keywords need to be added, they can be added accordingly in the preset keyword library.

[0184] It should be noted that, during implementation, the above-mentioned coordinate difference may correspond to the difference information of the position information in the aforementioned embodiment, the video sequence may correspond to the image frame sequence in the aforementioned embodiment, the single-frame feature extraction network 701 may correspond to the spatial feature extraction subnetwork in the aforementioned embodiment, the inter-frame feature fusion network 702 may correspond to the temporal feature extraction subnetwork in the aforementioned embodiment, the syllable feature layer 7031 may correspond to the syllable classification feature extraction subnetwork in the aforementioned embodiment, and the first linear layer 7032 may correspond to the classification network in the aforementioned embodiment.

[0185] In the field of human-computer interaction, the application of speech recognition still has certain limitations, such as in situations where there is loud noise or music volume, or where it is inconvenient to speak. In these situations, lip reading recognition can, to a certain extent, make up for the inconvenience caused by the limitations of speech recognition. Lip reading recognition can detect the keywords corresponding to the content spoken by the speaker within the speaking interval detected by lip movement recognition. For example, in the car cabin, speech recognition is the main means of human-computer interaction, but when the car is noisy on the highway or the music is playing loudly, speech recognition cannot accurately recognize the user's voice; or when someone in the car is sleeping, it is not convenient for the user to interact by voice. At this time, through lip reading recognition, the user only needs to use the mouth shape to simulate speaking, and the car computer can detect the user's instructions, thus completing the human-computer interaction.

[0186] Compared to related lip reading recognition technologies, the present embodiment utilizes key point recognition, consuming fewer computing resources and capable of learning inter-frame lip motion information, making deployment easier, more efficient, and more accurate. The image processing method provided in this embodiment supports lip reading recognition, supporting the recognition of 35 common keywords and achieving a recall rate of 81% while keeping the false alarm rate to less than one in a thousand.

[0187] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes the various units included and the various modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0188] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the image processing device 800 includes: a first acquisition module 810, a first recognition module 820, a first determination module 830 and a first matching module 840, wherein:

[0189] A first acquisition module 810 is configured to acquire a sequence of image frames containing a mouth object;

[0190] A first recognition module 820 is configured to extract mouth key point features from each image frame in the image frame sequence to obtain mouth key point features of each image frame;

[0191] A first determining module 830 is configured to generate a syllable classification feature based on the mouth key point features of a plurality of image frames in the image frame sequence; wherein the syllable classification feature represents the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence;

[0192] The first matching module 840 is configured to determine keywords matching the syllable classification feature in a preset keyword library.

[0193] In some embodiments, when the image frame sequence includes at least two frames of images, the first recognition module 820 includes: a first determination submodule, used to determine the position information of at least two mouth key points of the mouth object in each image frame; and a second determination submodule, used to determine, for each image frame in the image frame sequence, the mouth key point features corresponding to the image frame based on the position information of the mouth key points in the image frame and the adjacent frames of the image frame.

[0194] In some embodiments, the mouth key point features include inter-frame difference information and intra-frame difference information of each of the mouth key points; the second determination submodule includes: a first determination unit, for determining, for each of the mouth key points, the first height difference and / or first width difference of the mouth key point between the image frame and the adjacent frame according to the position information of the mouth key point in the image frame and the position information of the mouth key point in the adjacent image frame of the image frame, as the inter-frame difference information of the mouth key point; a second determination unit, for each of the mouth key points, for determining, according to the second height difference and / or second width difference between the mouth key point in the image frame and other mouth key points of the same mouth object, the intra-frame difference information of the mouth key point.

[0195] In some embodiments, the first determination module 830 includes: a first extraction submodule, used to perform spatial feature extraction on the mouth key point features of each of the image frames, respectively, to obtain the spatial features of the mouth object in each image frame; a second extraction submodule, used to perform temporal feature extraction on the spatial features of the mouth object in multiple image frames, to obtain the spatiotemporal features of the mouth object; and a third extraction submodule, used to perform syllable classification feature extraction based on the spatiotemporal features of the mouth object, to obtain the syllable classification features of the mouth object.

[0196] In some embodiments, the first extraction submodule includes: a first extraction unit, used to fuse the inter-frame difference information and intra-frame difference information of the multiple mouth key points of the mouth object to obtain the inter-frame difference features and intra-frame difference features of the mouth object in each image frame; a second extraction unit, used to fuse the inter-frame difference features and intra-frame difference features of the mouth object in multiple image frames to obtain the spatial features of the mouth object in each image frame.

[0197] In some embodiments, the first determination module 830 includes: a third determination submodule, which is used to use a trained syllable feature extraction network to process the mouth key point features of multiple image frames in the image frame sequence to obtain syllable classification features; the first matching module 840 includes: a first matching submodule, which is used to determine the keywords that match the syllable classification features in the preset keyword library, including: using a trained classification network to determine the keywords that match the syllable classification features in the preset keyword library.

[0198] In some embodiments, the first acquisition module 810 includes an interpolation submodule, which is used to: perform image interpolation on the acquired original image sequence containing the mouth object to obtain the image frame sequence; or, based on the mouth key points in the acquired original image sequence containing the mouth object, perform interpolation on the original image sequence to obtain the image frame sequence.

[0199] In some embodiments, the syllable feature extraction network includes a spatial feature extraction subnetwork, a temporal feature extraction subnetwork and a classification feature extraction subnetwork; the third determination submodule includes: a third extraction unit, used to use the spatial feature extraction subnetwork to perform spatial feature extraction on the mouth key point features of each image frame respectively, to obtain the spatial features of the mouth object in each image frame; a fourth extraction unit, used to use the temporal feature extraction subnetwork to perform temporal feature extraction on the spatial features of the mouth object in multiple image frames, to obtain the spatiotemporal features of the mouth object; a fifth extraction unit, used to use the classification feature extraction subnetwork to perform syllable classification feature extraction based on the spatiotemporal features of the mouth object, to obtain the syllable classification features of the mouth object.

[0200] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0201] Based on the foregoing embodiments, an embodiment of the present application provides a device for generating a lip reading recognition model, which includes the various units included and the various modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; during implementation, the processor can be a CPU, MPU, DSP or FPGA, etc.

[0202] Figure 9 A schematic diagram of the structure of a device for generating a lip reading recognition model provided in an embodiment of the present application is shown in FIG. Figure 9As shown, the apparatus 900 includes: a second acquisition module 910, a second identification module 920, a second matching module 930 and an update module 940, wherein:

[0203] A second acquisition module 910 is configured to acquire a sample image frame sequence containing a mouth object; wherein the sample image frame sequence is annotated with a keyword tag;

[0204] A second recognition module 920 is configured to extract mouth key point features from each sample image frame in the sample image frame sequence to obtain mouth key point features of each sample image frame;

[0205] A second matching module 930 is configured to generate a syllable classification feature based on the mouth key point features of a plurality of sample image frames in the sample image frame sequence using the model to be trained, and to determine a keyword matching the syllable classification feature in a preset keyword library; wherein the syllable classification feature represents the syllable category corresponding to the mouth shape of the mouth object in the sample image frame sequence;

[0206] The updating module 940 is configured to update the network parameters of the model at least once based on the determined keywords and keyword tags to obtain a trained lip reading recognition model.

[0207] In some embodiments, the model includes a syllable feature extraction network and a classification network; the second matching module 930 includes: a fourth determination submodule, used to use the feature extraction network to generate syllable classification features based on the mouth key point features of multiple sample image frames in the sample image frame sequence; a fifth determination submodule, used to use the classification network to determine keywords that match the syllable classification features in a preset keyword library.

[0208] In some embodiments, the feature extraction network includes a spatial feature extraction subnetwork, a temporal feature extraction subnetwork and a syllable classification feature extraction subnetwork; the fourth determination submodule includes: a sixth extraction unit, used to use the spatial feature extraction subnetwork to perform spatial feature extraction on the mouth key point features of each sample image frame respectively, to obtain the spatial features of the mouth object in each sample image frame; a seventh extraction unit, used to use the temporal feature extraction subnetwork to perform sample time feature extraction on the spatial features of the mouth object in multiple sample image frames, to obtain the spatiotemporal features of the mouth object; an eighth extraction unit, used to use the syllable classification feature extraction subnetwork to perform syllable classification feature extraction based on the spatiotemporal features of the mouth object, to obtain the syllable classification features of the mouth object.

[0209] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0210] An embodiment of the present application provides a vehicle, comprising:

[0211] an onboard camera for capturing a sequence of image frames containing a mouth object;

[0212] A vehicle computer is connected to the vehicle-mounted camera and is used to obtain an image frame sequence containing a mouth object from the vehicle-mounted camera; extract mouth key point features for each image frame in the image frame sequence to obtain mouth key point features of each image frame; generate syllable classification features based on the mouth key point features of multiple image frames in the image frame sequence; wherein the syllable classification features represent the syllable category corresponding to the mouth shape of the mouth object in the image frame sequence; and determine keywords that match the syllable classification features in a preset keyword library.

[0213] The description of the above vehicle embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the vehicle embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0214] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0215] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0216] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0217] An embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0218] An embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0219] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between the various embodiments, and their similarities or similarities can be referenced to each other. The descriptions of the above device, storage medium, computer program, and computer program product embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the description of the method embodiments of this application for understanding.

[0220] It should be noted that Figure 10 A schematic diagram of a hardware entity of a computer device in an embodiment of the present application is shown in FIG. Figure 10 As shown, the hardware entity of the computer device 1000 includes: a processor 1001, a communication interface 1002 and a memory 1003, wherein:

[0221] Processor 1001 generally controls the overall operation of computer device 1000 .

[0222] The communication interface 1002 enables the computer device to communicate with other terminals or servers through a network.

[0223] The memory 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or processed by the processor 1001 and various modules in the computer device 1000 (for example, image data, audio data, voice communication data, and video communication data). This can be implemented using flash memory (FLASH) or random access memory (RAM). Data can be transmitted between the processor 1001, the communication interface 1002, and the memory 1003 via the bus 1004.

[0224] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0225] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0227] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0228] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0229] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0230] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0231] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0232] The above is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Obtain a sequence of image frames containing a mouth object; Extracting mouth key point features from each image frame in the image frame sequence to obtain mouth key point features of each image frame; generating a syllable classification feature based on the mouth key point features of a plurality of image frames in the image frame sequence; wherein the syllable classification feature represents a syllable category corresponding to the mouth shape of the mouth object in the image frame sequence, the syllable category including at least one of a set number of preset syllable categories, the set number of preset syllable categories being obtained by classifying at least one syllable type based on the similarity of the mouth shapes; Determining keywords that match the syllable classification features in a preset keyword library; The step of generating syllable classification features based on the mouth key point features of the plurality of image frames in the image frame sequence includes: Performing spatial feature extraction on the mouth key point features of each image frame to obtain the spatial features of the mouth object in each image frame; Performing temporal feature extraction on the spatial features of the mouth object in the plurality of image frames to obtain spatiotemporal features of the mouth object; Syllable classification features are extracted based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

2. The method according to claim 1, characterized in that The extracting the mouth key point features of each image frame in the image frame sequence to obtain the mouth key point features of each image frame includes: Determine position information of at least two mouth key points of the mouth object in each image frame; For each image frame in the image frame sequence, a mouth key point feature corresponding to the image frame is determined based on position information of the mouth key points in the image frame and adjacent frames of the image frame.

3. The method according to claim 2, characterized in that The mouth key point features include inter-frame difference information and intra-frame difference information of each mouth key point; The determining, based on the position information of the mouth key points in the image frame and adjacent frames of the image frame, the mouth key point features corresponding to the image frame includes: For each of the mouth key points, determining, based on position information of the mouth key point in the image frame and position information of the mouth key point in an adjacent image frame of the image frame, a first height difference and / or a first width difference of the mouth key point between the image frame and the adjacent frame as inter-frame difference information of the mouth key point; For each of the mouth key points, intra-frame difference information of the mouth key point is determined according to a second height difference and / or a second width difference between the mouth key point and other mouth key points of the same mouth object in the image frame.

4. The method according to any one of claims 1 to 3, characterized in that The step of extracting spatial features of the mouth key point features of each image frame to obtain spatial features of the mouth object in each image frame includes: fusing inter-frame difference information and intra-frame difference information of a plurality of mouth key points of the mouth object to obtain inter-frame difference features and intra-frame difference features of the mouth object in each image frame; The inter-frame difference features and intra-frame difference features of the mouth object in the plurality of image frames are fused to obtain the spatial features of the mouth object in each image frame.

5. The method according to any one of claims 1 to 4, characterized in that Generating syllable classification features based on the mouth key point features of the plurality of image frames in the image frame sequence includes: processing the mouth key point features of the plurality of image frames in the image frame sequence using a trained syllable feature extraction network to obtain syllable classification features; The determining of the keywords matching the syllable classification features in the preset keyword library includes: using a trained classification network to determine the keywords matching the syllable classification features in the preset keyword library.

6. The method according to any one of claims 1 to 5, characterized in that The acquiring of a sequence of image frames containing a mouth object comprises: Performing image interpolation on the acquired original image sequence containing the mouth object to obtain the image frame sequence; or, Based on the acquired mouth key points in the original image sequence containing the mouth object, the original image sequence is interpolated to obtain the image frame sequence.

7. A method for generating a lip reading recognition model, characterized in that: The method comprises: Acquire a sample image frame sequence containing a mouth object; wherein the sample image frame sequence is annotated with a keyword label; Extracting mouth key point features from each sample image frame in the sample image frame sequence to obtain mouth key point features of each sample image frame; Using the model to be trained, generating a syllable classification feature based on the mouth key point features of multiple sample image frames in the sample image frame sequence, and determining a keyword matching the syllable classification feature in a preset keyword library; wherein the syllable classification feature represents the syllable category corresponding to the mouth shape of the mouth object in the sample image frame sequence, and the syllable category includes at least one of a set number of preset syllable categories, and the set number of preset syllable categories is obtained by classifying at least one syllable type according to the similarity of the mouth shapes; Based on the determined keywords and keyword tags, updating the network parameters of the model at least once to obtain a trained lip reading recognition model; The step of generating syllable classification features based on the mouth key point features of a plurality of sample image frames in the sample image frame sequence includes: Performing spatial feature extraction on the mouth key point features of each sample image frame to obtain the spatial features of the mouth object in each sample image frame; Performing sample time feature extraction on the spatial features of the mouth object in the plurality of sample image frames to obtain the spatiotemporal features of the mouth object; Syllable classification features are extracted based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

8. The method according to claim 7, characterized in that The model includes a syllable feature extraction network and a classification network; using the model to be trained, generating syllable classification features based on the mouth key point features of multiple sample image frames in the sample image frame sequence, and determining keywords matching the syllable classification features in a preset keyword library, including: generating syllable classification features based on the mouth key point features of a plurality of sample image frames in the sample image frame sequence using the syllable feature extraction network; The classification network is used to determine keywords that match the syllable classification features in a preset keyword library.

9. The method according to claim 8, characterized in that The syllable feature extraction network includes a spatial feature extraction subnetwork, a temporal feature extraction subnetwork and a syllable classification feature extraction subnetwork; The step of utilizing the syllable feature extraction network to generate syllable classification features based on the mouth key point features of a plurality of sample image frames in the sample image frame sequence includes: Using the spatial feature extraction subnetwork, perform spatial feature extraction on the mouth key point features of each sample image frame to obtain the spatial features of the mouth object in each sample image frame; Using the temporal feature extraction subnetwork, performing sample temporal feature extraction on the spatial features of the mouth object in the plurality of sample image frames to obtain the spatiotemporal features of the mouth object; The syllable classification feature extraction subnetwork is used to extract syllable classification features based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

10. An image processing device, characterized in that: include: A first acquisition module is used to acquire a sequence of image frames containing a mouth object; a first recognition module, configured to extract mouth key point features from each image frame in the image frame sequence to obtain mouth key point features of each image frame; a first determining module configured to generate a syllable classification feature based on the mouth key point features of a plurality of image frames in the image frame sequence; wherein the syllable classification feature represents a syllable category corresponding to a mouth shape of a mouth object in the image frame sequence, the syllable category including at least one of a set number of preset syllable categories, the set number of preset syllable categories being obtained by classifying at least one syllable type based on a degree of similarity in the mouth shapes; A first matching module is used to determine a keyword matching the syllable classification feature in a preset keyword library; Wherein, the first determining module is further configured to: Performing spatial feature extraction on the mouth key point features of each image frame to obtain the spatial features of the mouth object in each image frame; Performing temporal feature extraction on the spatial features of the mouth object in the plurality of image frames to obtain spatiotemporal features of the mouth object; Syllable classification features are extracted based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

11. A device for generating a lip reading recognition model, characterized in that: include: A second acquisition module is configured to acquire a sample image frame sequence containing a mouth object; wherein the sample image frame sequence is annotated with a keyword tag; a second recognition module, configured to extract mouth key point features from each sample image frame in the sample image frame sequence to obtain mouth key point features of each sample image frame; a second matching module, configured to generate, using the model to be trained, a syllable classification feature based on the mouth key point features of a plurality of sample image frames in the sample image frame sequence, and determine a keyword matching the syllable classification feature in a preset keyword library; wherein the syllable classification feature represents a syllable category corresponding to the mouth shape of the mouth object in the sample image frame sequence, the syllable category including at least one of a set number of preset syllable categories, the set number of preset syllable categories being obtained by classifying at least one syllable type based on the similarity of the mouth shapes; an updating module, configured to update the network parameters of the model at least once based on the determined keywords and keyword tags, to obtain a trained lip reading recognition model; Wherein, the second matching module is further used for: Performing spatial feature extraction on the mouth key point features of each sample image frame to obtain the spatial features of the mouth object in each sample image frame; Performing sample time feature extraction on the spatial features of the mouth object in the plurality of sample image frames to obtain the spatiotemporal features of the mouth object; Syllable classification features are extracted based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A vehicle comprising: an onboard camera for capturing a sequence of image frames containing a mouth object; A vehicle computer, connected to the vehicle-mounted camera, is used to obtain a sequence of image frames containing a mouth object from the vehicle-mounted camera; extract mouth key point features from each image frame in the sequence of image frames to obtain mouth key point features of each image frame; generating a syllable classification feature based on the mouth key point features of a plurality of image frames in the image frame sequence; wherein the syllable classification feature represents a syllable category corresponding to the mouth shape of the mouth object in the image frame sequence, the syllable category including at least one of a set number of preset syllable categories, the set number of preset syllable categories being obtained by classifying at least one syllable type based on the similarity of the mouth shapes; and determining a keyword matching the syllable classification feature in a preset keyword library; The vehicle computer is also used for: Performing spatial feature extraction on the mouth key point features of each image frame to obtain the spatial features of the mouth object in each image frame; Performing temporal feature extraction on the spatial features of the mouth object in the plurality of image frames to obtain spatiotemporal features of the mouth object; Syllable classification features are extracted based on the spatiotemporal features of the mouth object to obtain syllable classification features of the mouth object.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images