Lip language recognition method, sample labeling method, model training method and device, equipment, and storage medium

Abstract

The application relates to a lip language recognition method, a sample labeling method, a model training method and device, equipment and a storage medium. The application relates to the lip language recognition technology in the field of artificial intelligence. In the embodiment of the application, the lip language video clips of a user are labeled by a voice recognition text of the user, and then the lip language recognition sample of the user is automatically obtained. The lip language recognition model is trained by using the lip language recognition sample, and the lip language recognition of the user is performed by using the lip language recognition model. The automatic labeling of the lip language recognition sample, the self-supervision active learning and the individualization of the lip language recognition model are realized. The lip language recognition accuracy is effectively improved. The whole process does not require the participation of the user, and the user experience is improved.

Description

[0001] This application claims priority to Chinese Patent Application No. 202110643378.3, filed on June 9, 2021, entitled "Lip Reading Recognition Method, Sample Annotation Method, Model Training Method and Apparatus, Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to lip reading technology in the field of artificial intelligence, and in particular to a lip reading method, sample annotation method, model training method, device, equipment, and storage medium. Background Technology

[0003] Lip reading combines computer vision and natural language processing, allowing the interpretation of speech solely through the speaker's facial features in a video. It has proven effective in areas such as intelligent human-computer interaction and audio corruption, and holds significant practical importance for capturing and recognizing target language in the context of hearing impairment and aphasia, as well as in noisy environments.

[0004] With the rapid development of deep learning and large-scale datasets, lip-reading technology based on lip-reading models has significantly improved performance compared to traditional lip-reading techniques in word or sentence-level recognition tasks. However, due to differences in hardware devices, as well as variations in the static and dynamic features of the lips and language habits of different people, the accuracy of lip-reading models varies considerably among different individuals. Summary of the Invention

[0005] In view of the above-mentioned problems of the prior art, this application provides a lip reading recognition method, sample annotation method, model training method, apparatus, device, and storage medium, which can automatically complete the annotation of lip reading recognition samples, improve the recognition accuracy of the lip reading recognition model, and thus improve the lip reading recognition accuracy of various types of people.

[0006] To achieve the above objectives, the first aspect of this application provides a sample annotation method applied to electronic devices, the sample annotation method comprising:

[0007] The system captures the user's video and audio while the user is speaking into the electronic device.

[0008] Extract lip movement video segments from the video and speech segments from the audio;

[0009] Select the lip movement video clip that matches the voice clip;

[0010] The lip movement video segments are labeled using the speech recognition text of the speech segments as tags to obtain the user's lip reading samples.

[0011] By simultaneously acquiring video and audio of the user speaking, and using the speech recognition results of speech segments in the audio as labels for corresponding lip-reading video segments in the video, lip-reading samples can be automatically labeled without user intervention. This improves the efficiency and accuracy of sample labeling while enabling personalized labeling.

[0012] As one possible implementation of the first aspect, the method further includes one or more of the following: detecting the volume of ambient noise; obtaining the user's wake-up voice confidence level; detecting whether a face or mouth is contained within the field of view of the camera; obtaining the speaker's position in the video and the sound source localization direction of the audio; wherein obtaining the user's video and audio specifically includes: obtaining the user's video and audio when the volume of ambient noise is equal to or less than a preset noise threshold, the wake-up voice confidence level is greater than or equal to a preset first confidence threshold, a face or mouth is contained within the field of view of the camera, and / or the speaker's position in the video and the sound source localization direction of the audio match.

[0013] This avoids the problem of inaccurate sample labels caused by poor voice quality, environmental noise, poor video quality, and / or audio-video asynchrony, and can effectively improve the accuracy of sample labeling.

[0014] As one possible implementation of the first aspect, extracting lip-movement video segments from the video and speech segments from the audio includes: performing endpoint detection and segmentation on the video using a lip-movement voice interval detection method to obtain lip-movement video segments and their voice intervals; and / or, performing endpoint detection and segmentation on the audio using a speech voice interval detection method to obtain speech segments and their voice intervals. Thus, by segmenting the video and audio using endpoint detection, the corresponding voice intervals can be obtained simultaneously to determine the matching relationship between speech segments and lip-movement video segments; in other words, to find speech segments and lip-movement video segments corresponding to the same spoken content.

[0015] As one possible implementation of the first aspect, selecting lip-movement video segments that match speech segments includes: determining the overlap length between the speech segment and the lip-movement video segment in the time dimension based on the human voice interval of the speech segment and the human voice interval of the lip-movement video segment; when the overlap length between the speech segment and the lip-movement video segment in the time dimension is greater than or equal to a predetermined duration threshold, the speech segment and the lip-movement video segment are considered a match. Therefore, determining the overlap length between the speech segment and the lip-movement video segment in the time dimension through the human voice interval allows for efficient, fast, and relatively accurate matching of lip-movement video segments for each speech segment; that is, it allows for efficient, fast, and relatively accurate identification of speech segments and lip-movement video segments corresponding to the same spoken content.

[0016] As one possible implementation of the first aspect, the method further includes: selecting lip movement video segments with a lip-reading recognition confidence level less than a preset second confidence threshold from among the lip movement video segments matched with the speech segment. The lip-reading recognition confidence level is obtained by performing lip-reading recognition on the lip movement video segments based on a pre-obtained lip-reading recognition model. Thus, the "most uncertain" or "most information-carrying" lip movement video segments can be selected as lip-reading recognition samples, providing supervised samples for subsequent transfer learning and iterative optimization of the lip-reading recognition model.

[0017] As one possible implementation of the first aspect, the lip movement video clip includes a sequence of lip movement images, wherein the image frames in the sequence are images of the lip region.

[0018] The second aspect of this application provides a model training method for use in electronic devices, comprising: updating the parameters of a lip-reading recognition model using lip-reading recognition samples obtained by the sample annotation method of the first aspect. Thus, by using a speech modality to assist a visual modality in automatic annotation and model optimization, self-supervised, personalized, and active learning of the lip-reading recognition model is achieved, improving the recognition accuracy and personalization of the lip-reading recognition model. Furthermore, the entire process requires no user intervention, enhancing the user experience.

[0019] As one possible implementation of the second aspect, the lip-reading model includes a general feature layer and a trainable layer. The parameters of the lip-reading model include trainable layer parameters and general feature layer parameters. Updating the parameters of the lip-reading model specifically involves updating the trainable layer parameters. Therefore, by updating the trainable layer parameters of the lip-reading model for each user, not only is the training efficiency higher, but the amount of lip-reading model parameter data for each user is relatively small, which helps save hardware resources.

[0020] As a possible implementation of the second aspect, the model training method also includes: associating the trainable layer parameters with the user's preset information and then saving them. By associating the trainable layer parameters of the lip-reading model with the user's preset information, the user's lip-reading model parameters can be easily and quickly found using the user's preset information.

[0021] As a possible implementation of the second aspect, the model training method further includes: storing the user's preset information in a registered information database; and storing the trainable layer parameters associated with the preset information in a lip-reading model library. This allows for convenient confirmation of the existence of the user's lip-reading model parameters through the user's preset information, and also enables quick retrieval of the user's lip-reading model parameters using the user's preset information.

[0022] As a possible implementation of the second aspect, before updating the parameters of the lip-reading model, the method further includes: adjusting the parameter update rate of the lip-reading model by comparing the lip-reading text of the lip-reading sample with the label of the lip-reading sample, to obtain the parameter update rate of the corresponding lip-reading sample; wherein, the lip-reading text is obtained by the lip-reading model performing lip-reading on the lip-reading sample; updating the parameters of the lip-reading model specifically includes: updating the parameters of the lip-reading model using the parameter update rate of the lip-reading sample and the corresponding lip-reading sample. This can improve the optimization efficiency of the lip-reading model and reduce hardware resource consumption.

[0023] The third aspect of this application provides a lip-reading recognition method applied to an electronic device, comprising: acquiring a video of the user when the user speaks into the electronic device; extracting lip movement video segments from the video; running the lip-reading recognition model based on the parameters of the lip-reading recognition model obtained by the model training method of the second aspect, performing lip-reading recognition on the lip movement video segments, and obtaining lip-reading recognized text. Thus, lip-reading recognition for various types of people can be efficiently completed through electronic devices, while improving the accuracy of lip-reading recognition for various types of people.

[0024] As a possible implementation of the third aspect, the lip-reading model includes a general feature layer and a trainable layer. The parameters of the lip-reading model include trainable layer parameters and general feature layer parameters. Running the lip-reading model based on the parameters obtained from the model training method to perform lip-reading on the lip-movement video segment includes: acquiring the user's preset information; acquiring the trainable layer parameters associated with the preset information; loading the trainable layer parameters and pre-configured general feature layer parameters to run the lip-reading model to perform lip-reading on the lip-movement video segment. Thus, the speaker's trainable layer parameters can be quickly found through the speaker's preset information. Using these trainable layer parameters and general feature layer parameters shared by various users to run the lip-reading model to perform lip-reading on the speaker is equivalent to using a user-customized lip-reading model to perform lip-reading on the user, thereby effectively improving the individual-specific lip-reading accuracy in a specific domain.

[0025] As a possible implementation of the third aspect, the preset information includes a face ID; obtaining the user's preset information specifically includes: performing face recognition on the video to obtain the user's facial feature data, and querying the face ID corresponding to the facial feature data from the registered face database. Therefore, the user's preset information can be obtained through video, which helps improve the processing efficiency of lip reading recognition and further improves the accuracy of individual lip reading recognition.

[0026] As a possible implementation of the third aspect, running the lip-reading recognition model based on the parameters of the lip-reading recognition model obtained from the model training method to perform lip-reading recognition on the lip movement video segment further includes: when the trainable layer parameters associated with preset information are not available, loading locally stored trainable layer parameters and general feature layer parameters to run the lip-reading recognition model to perform lip-reading recognition on the lip movement video segment. Therefore, lip-reading recognition can be completed using general lip-reading recognition model parameters even when the user's trainable layer parameters are not available, which is equivalent to using a general lip-reading recognition model to perform lip-reading recognition on the user, thus enabling efficient lip-reading recognition for various types of people through electronic devices.

[0027] A fourth aspect of this application provides a lip-reading recognition device for use in electronic devices, the lip-reading recognition device comprising:

[0028] The video acquisition unit is configured to acquire the user's video while the user is speaking into the electronic device;

[0029] The audio acquisition unit is configured to acquire the user's audio while the user is speaking into the electronic device;

[0030] The lip movement extraction unit is configured to extract lip movement video segments from a video.

[0031] The speech extraction unit is configured to extract speech segments from audio.

[0032] The selection unit is configured to select a lip movement video segment that matches the speech segment;

[0033] The annotation unit is configured to use the speech recognition text of the speech segment as a label to annotate the lip movement video segment, thereby obtaining the user's lip reading recognition sample.

[0034] As a possible implementation of the fourth aspect, the lip-reading device also includes one or more of the following:

[0035] A noise detection unit is configured to detect the volume of ambient noise.

[0036] The wake-up voice confidence acquisition unit is configured to acquire the user's wake-up voice confidence.

[0037] The face detection unit is configured to detect whether a face or mouth is present within the field of view of the camera;

[0038] The positioning unit is configured to acquire the speaker's position in the video and the sound source localization direction in the audio.

[0039] The video acquisition unit is specifically configured to acquire the user's video when: the ambient noise volume is equal to or less than a preset noise threshold; the wake-up voice confidence level is greater than or equal to a preset first confidence threshold; the camera's field of view contains a face or mouth; and / or the speaker's position in the video matches the sound source localization direction in the audio; and / or...

[0040] The audio acquisition unit is specifically configured to acquire the user's audio when the ambient noise volume is equal to or less than a preset noise threshold, the wake-up voice confidence is greater than or equal to a preset first confidence threshold, the camera's field of view contains a face or mouth, and / or the speaker's position in the video matches the sound source localization direction of the audio.

[0041] As one possible implementation of the fourth aspect, the lip movement extraction unit is specifically configured to: perform endpoint detection and segmentation on the video using a lip movement voice interval detection method to obtain lip movement video segments and voice intervals of the lip movement video segments; and / or, the speech extraction unit is specifically configured to: perform endpoint detection and segmentation on the audio using a speech voice interval detection method to obtain speech segments and voice intervals of the speech segments.

[0042] As a possible implementation of the fourth aspect, the selection unit is specifically configured as follows: based on the human voice interval of the speech segment and the human voice interval of the lip-moving video segment, the overlap length between the speech segment and the lip-moving video segment in the time dimension is determined; when the overlap length between the speech segment and the lip-moving video segment in the time dimension is greater than or equal to a predetermined duration threshold, the speech segment and the lip-moving video segment are matched.

[0043] As a possible implementation of the fourth aspect, the selection unit is further configured to: select lip movement video segments whose lip reading confidence is less than a preset second confidence threshold from among the lip movement video segments that match the speech segment, wherein the lip reading confidence is obtained by performing lip reading recognition on the lip movement video segments according to a pre-obtained lip reading recognition model.

[0044] As one possible implementation of the fourth aspect, the lip movement video clip includes a sequence of lip movement images, in which the image frames are images of the lip region.

[0045] As a possible implementation of the fourth aspect, the lip reading device further includes: a parameter update unit configured to update the parameters of the lip reading model using lip reading samples obtained by the annotation unit.

[0046] As one possible implementation of the fourth aspect, the lip reading recognition model includes a general feature layer and a trainable layer, and the parameters of the lip reading recognition model include trainable layer parameters and general feature layer parameters; the parameter update unit is specifically configured to update the trainable layer parameters of the lip reading recognition model.

[0047] As a possible implementation of the fourth aspect, the lip-reading device also includes a storage unit configured to associate trainable layer parameters with user preset information and save them.

[0048] As one possible implementation of the fourth aspect, the storage unit is specifically configured to: store the user's preset information in the registered information database; and store the trainable layer parameters associated with the preset information in the lip reading model library.

[0049] As one possible implementation of the fourth aspect, the parameter update unit is specifically configured to adjust the parameter update rate of the lip-reading model by comparing the lip-reading text of the lip-reading sample with the label of the lip-reading sample to obtain the parameter update rate of the corresponding lip-reading sample; and update the parameters of the lip-reading model using the lip-reading sample and the parameter update rate of the corresponding lip-reading sample; wherein, the lip-reading text is obtained by the lip-reading model performing lip-reading on the lip-reading sample.

[0050] As one possible implementation of the fourth aspect, the video acquisition unit is further configured to acquire the user's video when the user speaks to the electronic device; the lip reading device also includes: a lip reading recognition unit configured to run the lip reading recognition model according to the parameters of the lip reading recognition model updated by the parameter update unit to perform lip reading recognition on the lip movement video segment and obtain lip reading recognition text.

[0051] As a possible implementation of the fourth aspect, the lip-reading recognition model includes a general feature layer and a trainable layer. The parameters of the lip-reading recognition model include trainable layer parameters and general feature layer parameters. The lip-reading recognition device also includes: a preset information acquisition unit configured to acquire preset information from the user; the lip-reading recognition unit is specifically configured to: acquire the user's preset information; acquire trainable layer parameters associated with the preset information; load the trainable layer parameters and pre-configured general feature layer parameters to run the lip-reading recognition model to perform lip-reading recognition on lip movement video segments.

[0052] As one possible implementation of the fourth aspect, the preset information includes a face ID; the preset information acquisition unit is specifically configured to: perform face recognition on image frames in the video to obtain the user's face feature data, and query the registered face database for the face ID corresponding to the face feature data.

[0053] The fifth aspect of this application provides an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to perform the sample annotation method of the first aspect, the model training method of the second aspect, and / or the lip reading recognition method of the third aspect.

[0054] The sixth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, characterized in that, when executed by a computer, the program instructions cause the computer to perform the sample annotation method of the first aspect, the model training method of the second aspect, and / or the lip reading recognition method of the third aspect.

[0055] This application embodiment captures both video and audio of the user speaking simultaneously, uses the speech recognition results of speech segments in the audio as labels for corresponding lip-reading video segments in the video, thereby automatically obtaining the user's lip-reading recognition samples. The lip-reading recognition model is then optimized or trained using the user's lip-reading recognition samples, thereby achieving user customization of the lip-reading recognition model. Finally, the lip-reading recognition model is used to perform lip-reading recognition on the user, effectively improving the lip-reading recognition accuracy for specific users or specific scenarios. Attached Figure Description

[0056] The following description, with reference to the accompanying drawings, further illustrates the various features of this application and the relationships between them. The drawings are exemplary; some features are not shown to scale, and some drawings may omit conventional features in the field of this application that are not essential to it, or additional features that are not essential to this application may be shown. The combination of features shown in the drawings is not intended to limit this application. Furthermore, throughout this specification, the same reference numerals refer to the same things. Specific descriptions of the drawings are as follows:

[0057] Figure 1 This is a flowchart illustrating the sample annotation method provided in the embodiments of this application.

[0058] Figure 2 This is a schematic diagram showing the overlap of lip-movement video clips and speech clips in the time dimension in an embodiment of this application.

[0059] Figure 3 This is a flowchart illustrating the model training method provided in the embodiments of this application.

[0060] Figure 4 This is a flowchart illustrating the lip reading recognition method provided in the embodiments of this application.

[0061] Figure 5 This is a schematic flowchart of the lip reading recognition device provided in the embodiments of this application.

[0062] Figure 6 This is a schematic diagram illustrating the sample labeling and model training process in an exemplary application scenario of Embodiment 1 of this application.

[0063] Figure 7 This is a schematic diagram illustrating an exemplary implementation process of lip reading recognition, model training, and sample annotation in the embodiments of this application.

[0064] Figure 8This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0065] Figure 9 This is an exemplary structural diagram of the electronic device provided in the embodiments of this application.

[0066] Figure 10 This is a schematic diagram of an exemplary software architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] The terms "first, second, third, etc." or similar terms such as module A, module B, module C, etc., used in the specification and claims are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that a specific order or sequence may be interchanged where permitted so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0068] In the following description, the labels of the steps, such as S110, S120, etc., do not necessarily mean that the steps will be executed in this way. The order of the steps can be interchanged or executed simultaneously if permitted.

[0069] The term "comprising" as used in the specification and claims should not be construed as limiting itself to what follows; it does not exclude other elements or steps. Therefore, it should be interpreted as specifying the presence of the mentioned feature, integral, step, or component, but does not exclude the presence or addition of one or more other features, integrals, steps, or components, or groups thereof. Thus, the statement "device comprising means A and B" should not be limited to a device consisting solely of components A and B.

[0070] The terms "an embodiment" or "an embodiment" as used in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment of this application. Therefore, the terms "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. Furthermore, in one or more embodiments, the particular features, structures, or characteristics can be combined in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. In case of any inconsistency, the meaning set forth in this specification or derived from the content described herein shall prevail. Furthermore, the terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0072] In order to accurately describe the technical content of this application and to accurately understand this application, the following explanations or definitions of the terms used in this specification are given before describing the specific embodiments.

[0073] Automatic Speech Recognition (ASR) models are machine learning models that can recognize speech as text.

[0074] Voice Activity Detection (VAD) models use endpoint detection algorithms to identify the start and end points of human speech, thereby segmenting speech or video into multiple continuous segments and distinguishing between human and non-human speech segments. In simpler terms, endpoint detection aims to accurately locate the start and end points of human speech in noisy speech or video, finding a segment of speech or video that truly contains human voice content.

[0075] Sound source localization technology is a technique that uses sound source localization algorithms to determine the direction or location of an object (e.g., a speaker). In the embodiments of this application, the sound source localization technology employs a microphone array-based sound source localization algorithm, which may be a beamforming-based sound source localization algorithm, a high-resolution spectrum estimation-based sound source localization algorithm, or a sound source localization algorithm based on time delay difference of arrival.

[0076] Currently, lip-reading models typically employ supervised deep learning networks. To adapt these models to specific scenarios or user habits, it's often necessary to collect labeled samples (i.e., annotated samples) for those scenarios or users to fine-tune the model. Only with a large number of labeled samples can the model's performance be effectively improved, resolving the issue of low accuracy. However, current lip-reading model samples are usually collected and labeled manually. Collecting samples for specific scenarios or users is difficult, time-consuming, and inaccurate, making model optimization challenging and hindering performance improvement. Furthermore, pre-collecting large numbers of labeled samples compromises user privacy.

[0077] In view of this, embodiments of this application provide the following lip reading recognition methods, sample recognition methods, model training methods and apparatuses, devices, and computer-readable storage media. Embodiments of this application simultaneously capture video and audio of a user speaking, and use the speech recognition results of speech segments in the audio as labels for corresponding lip reading video segments in the video, thereby automatically obtaining lip reading samples of the user. The lip reading recognition model is optimized or trained using the user's lip reading samples, thereby realizing user customization of the lip reading recognition model. The lip reading recognition model is then used to perform lip reading recognition on the user, ultimately effectively improving the accuracy of lip reading recognition for specific users or specific scenarios.

[0078] This application embodiment can automatically annotate lip-reading samples without user intervention, achieving high efficiency and accuracy. It can also adapt to different scenarios and users to generate specific lip-reading samples, realizing personalized sample annotation. Furthermore, the entire process of this application embodiment is seamless for the user, requiring no pre-labeling for specific users or asking experts to determine the labels, making it more user-friendly and effectively improving the user experience. Through automatic sample annotation, this application embodiment achieves self-supervised active learning and personalization of the lip-reading model, effectively improving the customization of the lip-reading model and its recognition accuracy for specific users or scenarios, thereby increasing the accuracy of lip-reading recognition across various scenarios and for various types of people.

[0079] The embodiments of this application are applicable to various scenarios requiring lip-reading. Specifically, the embodiments of this application are applicable to various scenarios where a person speaks to an electronic device. For example, waking up an electronic device by speaking, making a call using an electronic device, playing interactive games using an electronic device, doing housework or other daily activities using an electronic device, controlling an electronic device by speaking to it (e.g., playing media content), or other similar scenarios.

[0080] The embodiments of this application can be applied to various electronic devices. For specific details about the electronic devices, please refer to the relevant description below, which will not be repeated here.

[0081] The methods described in this application embodiment can be implemented independently by an electronic device or by a system including an electronic device and a cloud server. In some embodiments, sample annotation, model training, and lip reading recognition in this application embodiment can all be completed by an electronic device. This eliminates the need to upload user lip reading samples for lip reading model training and lip reading based on the lip reading model, ensuring data integrity and protecting user privacy. In some embodiments, sample annotation and lip reading recognition in this application embodiment can be completed by an electronic device. The electronic device can provide locally obtained lip reading samples to the cloud server, which then updates the parameters of the lip reading model and sends the parameters back to the electronic device. This allows computationally complex and data-intensive processes such as model training to be handled by a powerful cloud server, improving processing efficiency and reducing the resource consumption of the electronic device.

[0082] The specific implementation methods of the embodiments of this application will be described in detail below.

[0083] Figure 1 A schematic flowchart of the sample annotation method provided in this application embodiment is shown. The sample annotation method of this application embodiment can be executed by an electronic device; technical details regarding the electronic device can be found in the relevant description below. See also Figure 1 As shown, the sample annotation method in this application embodiment may include the following steps:

[0084] In step S110, while the user is speaking into the electronic device, the user's video and audio are acquired.

[0085] Video can be captured using the camera of an electronic device. To avoid missing keyframes in the captured video, the user's mouth or entire face needs to be within the camera's field of view. In some embodiments, to facilitate the simultaneous acquisition of the user's preset information through video, it is preferable to include the user's entire face within the camera's field of view. When the user speaks into the electronic device, the electronic device detects that the user is speaking and begins capturing the user's video. The video includes the user's mouth movements during speaking; if no lip movements are detected, the acquisition of both video and audio can be stopped. Here, the detection of whether the user is speaking can be achieved by performing lip movement feature detection on the image captured by the camera that includes the user's mouth or face.

[0086] Audio can be captured via the microphone of an electronic device. When a user speaks into the electronic device, the device detects the user's speech and controls the microphone to capture the user's audio, which must contain at least the user's spoken words. If no lip movement is detected, audio capture can be stopped.

[0087] The specific format and length of the video and audio are not limited in this application embodiment. Audio can be short or long; for example, it can be a 30-second audio file, an audio file containing a single sentence, an audio file lasting ten minutes or more, an audio file containing a paragraph, or an audio file containing an article. Video can be short or long; for example, it can be a 30-second video, a video containing all mouth movements of a single sentence, a video lasting ten minutes or more, a video containing a paragraph, or a video containing an article.

[0088] For example, in scenarios involving "waking up an electronic device by speaking" or "controlling an electronic device by speaking," the audio could be an audio message containing a wake-up word or a voice command, and the video could be a sequence of mouth movement images during the speaking of the wake-up word or the speaking of the voice command. In scenarios involving "making a call using an electronic device," the audio could be the complete audio of a call, and the video could be a video of mouth movements during a call.

[0089] In some embodiments, the video and audio collected in this step can be of fixed length. Here, the fixed length of audio and video can be flexibly set according to various factors such as the specific application scenario, the processing power of the electronic device, the limitations of the VAD model, the limitations of the ASR model, and user needs. In this way, after collecting a certain length of video and a corresponding length of audio, the process jumps to the next step for processing. Thus, audio and video can be collected in real time while the user is speaking, and lip-reading samples can be generated in real time. Furthermore, it facilitates parallel processing of multiple video and audio segments, which helps improve processing efficiency, save time, and enhance the utilization of the electronic device's hardware resources.

[0090] When the user's voice quality is poor (e.g., the voice is too low, the speech is unclear, or the recognized text quality is poor), the accuracy of the voice segment recognition will be affected, resulting in inaccurate voice recognition results, which may reduce the accuracy of the labels in the lip-reading samples. To avoid this problem, before step S120 or step S110, the process may include: obtaining the user's wake-up voice confidence score. This allows comparing the wake-up voice confidence score with a preset first confidence threshold. If the wake-up voice confidence score is greater than or equal to the first confidence threshold, step S110 is executed to obtain the user's video and audio. If the wake-up voice confidence score is less than the first confidence threshold, it indicates that the user's current voice quality is poor, and the labeling process for the lip-reading samples may not be performed, i.e., step S110 may not be executed, or the video and its related audio may be discarded directly, without executing step S120 and subsequent processing. Therefore, by setting the confidence level of the wake-up speech and a threshold, the problem of low label accuracy caused by poor speech quality can be avoided, which can effectively improve the accuracy of sample labeling. Moreover, it is easy to implement and can be flexibly adjusted according to the actual situation.

[0091] A wake-up voice is the voice uttered by a user to wake up an electronic device. The confidence level of a wake-up voice is obtained through speech recognition. When waking up an electronic device with a wake-up voice, the device needs to perform speech recognition to obtain the content and confidence level of the wake-up voice. Typically, the electronic device is woken up when the confidence level of the wake-up voice is greater than or equal to a preset wake-up threshold and the content of the wake-up voice contains a preset wake-up word or phrase.

[0092] Generally, higher speech confidence indicates better speech quality. In some embodiments, the first confidence threshold can be a value greater than the wake-up threshold. This allows for sample annotation in this application embodiment to be performed when the user's speech quality is relatively good, thereby obtaining lip-reading recognition samples with high quality and accurate labels. For example, the wake-up confidence threshold is typically set to 0.5 or a lower value, while the first confidence threshold in this application embodiment can be set to a higher value such as 0.7, 0.8, or 0.85. In specific applications, the specific value of the first confidence threshold can be flexibly set according to one or more factors such as the specific application scenario, the accuracy of speech endpoint detection, the accuracy of speech recognition, and user needs. This application embodiment does not limit the specific configuration and determination method of the first confidence threshold.

[0093] The audio may contain environmental noise. To avoid environmental noise affecting the recognition accuracy of speech segments and thus reducing the accuracy of sample labels, the method may include detecting the volume of environmental noise before step S120 or step S110. This allows comparing the volume of the environmental noise with a preset noise threshold. If the environmental noise is less than or equal to the threshold, step S110 is executed to acquire the user's video and audio. If the environmental noise is greater than the threshold, it indicates that the environmental noise may interfere with the speech recognition of the speech segment. In this case, the labeling process for the lip-reading samples can be omitted, i.e., step S110 can be skipped, or the video segment and its associated audio can be discarded directly, without executing step S120 and subsequent processing. Therefore, by measuring the volume of environmental noise and setting a threshold, the impact of environmental noise on the speech recognition results of speech segments can be avoided, thereby improving the accuracy of labels in lip-reading samples. This method is easy to implement and can be flexibly adjusted according to actual conditions.

[0094] Ambient noise can be detected by a decibel meter built into the electronic device or by an external volume detector. In some embodiments, ambient noise can be detected during a predetermined period of time (e.g., 1 second, 2 seconds, 0.5 seconds) before, before, or after step S110, so that the detected ambient noise volume is closest to the ambient noise volume when the user speaks, thus achieving higher accuracy.

[0095] In practical applications, the noise threshold for environmental noise can be flexibly set based on one or more factors, such as the specific application scenario, the accuracy of voice endpoint detection, the accuracy of speech recognition in step S140, and user needs. In some embodiments, the noise threshold for environmental noise can be an empirical value, obtained by analyzing and statistically analyzing the speech recognition results of speech segments. In some embodiments, the noise threshold can be a dynamic value or a fixed value set manually. This application does not limit the specific configuration and determination method of the noise threshold.

[0096] To avoid missing keyframes in the video (e.g., an image frame of a key lip movement during a user's speech) and ensure that the video includes the user's face or mouth, before step S110 or S120, the process may further include: detecting whether a face or mouth is present within the camera's field of view, with the camera used to capture the video in step S110. If the face or mouth is not within the camera's field of view, it may cause the loss of keyframes. In this case, the sample annotation of this embodiment may not be performed. If the face or mouth is within the camera's field of view, the sample annotation of this embodiment can continue, and step S110 and subsequent steps can be executed.

[0097] Various applicable methods can be used to detect whether a face or mouth is present within the field of view of a camera. In some embodiments, this can be achieved by detecting whether the image captured by the camera contains facial or mouth features. If facial features are present, it indicates that a face or mouth is present within the camera's field of view; if mouth features are present, it indicates that a mouth is present within the camera's field of view. Here, facial and mouth features can be detected using general facial recognition algorithms, mouth detection algorithms, or pre-trained neural network models.

[0098] Optionally, to facilitate obtaining the user's preset information (e.g., face ID hereinafter) through the video in step S110, the face can be included within the field of view of the camera.

[0099] To ensure that the speaker in the video and the audio source are the same in step S110, before step S110 or S120, the method may further include: obtaining the speaker's position in the video and the sound source localization direction of the audio. Here, the sound source localization direction can indicate the speaker's orientation relative to the electronic device (relative to the microphone in the electronic device), and the speaker's position can indicate the speaker's position relative to the electronic device (e.g., relative to the camera in the electronic device). Therefore, the speaker's position can be compared with the sound source localization direction. If the speaker's position in the video and the sound source localization direction of the audio match, it indicates that the speaker in the video and the audio source are the same. In this case, the sample annotation of this embodiment can continue, i.e., step S110 can be executed to obtain the user's video and audio. If the speaker's position in the video and the sound source localization direction of the audio do not match, it indicates that the speaker in the video and the audio source are different. In this case, the sample annotation of this embodiment can be omitted, i.e., step S110 can be omitted, or the video segment and its associated audio can be discarded directly, and step S120 and subsequent processing can be skipped. In this embodiment of the application, sample annotation is performed under the premise that the speaker and the audio source are the same in the video, which can improve the accuracy of lip reading samples.

[0100] The direction of sound source localization in audio can be obtained using sound source localization technology. The speaker's location in a video can be, but is not limited to, the location of a face or mouth in the video, and can be obtained through face recognition in the video using algorithms such as human feature detection and image recognition algorithms.

[0101] Step S120: Extract lip movement video segments from the video and speech segments from the audio.

[0102] Before step S120, the process may further include: segmenting the video. Specifically, this segmentation may include: detecting and extracting the lip region of image frames within the mouth movement period in the video to form one or more mouth movement videos. In practical applications, various applicable methods can be used to implement this preprocessing, such as clustering, neural network-based feature extraction models, etc. This reduces the size of image frames in the video, lowers computational complexity and data volume, improves processing efficiency, and saves hardware resources.

[0103] Specifically, lip movement feature detection can be performed on each image frame in the video. Based on the results of the lip movement feature detection, the lip region of each image frame within the mouth movement period is extracted, taking each face region as a unit, to form a mouth movement video for each person. For example, the image frame size in the original video may be 1980*1024, but after cropping, the image frame size in the resulting mouth movement video can be 112*112, showing a significant reduction in image data.

[0104] In step S120, the video (e.g., the segmented mouth movement video) can be segmented and endpoint detected using a lip movement voice interval detection method to obtain lip movement video segments and their voice intervals. Specifically, the segmented mouth movement video can be segmented and endpoint detected using a lip movement VAD model or VAD algorithm to obtain lip movement video segments and their VAD values ​​(i.e., the voice intervals of the lip movement video segments). Here, the VAD value of a lip movement video segment indicates the start and end points of the segment, and the image frames in the segment are lip region images.

[0105] In step S120, the audio can be segmented and its endpoints detected using a speech voice interval detection method to obtain speech segments and their voice intervals. Specifically, the audio can be segmented and its endpoints detected using a speech VAD model or VAD algorithm to obtain speech segments and their VAD values ​​(i.e., the voice intervals of the speech segments). Here, the VAD value of a speech segment can indicate the start and end points of the speech segment.

[0106] In the embodiments of this application, the lip-movement VAD model or VAD algorithm, and the speech VAD model or VAD algorithm for endpoint detection and segmentation of audio, can employ endpoint detection methods based on short-time energy and zero-crossing rate, methods for classifying speech and non-speech based on neural network models, or any other applicable endpoint detection methods. This application does not limit its scope in this regard.

[0107] In this embodiment, the lengths of the lip movement video clips and speech clips can be determined by the VAD model or VAD algorithm used. For example, the lip movement video clips may include, but are not limited to, a sequence of lip movement images for a sentence or a sequence of lip movement images for a word, and the speech clips may include, but are not limited to, a speech segment for a sentence or the speech of a word. The lengths of the lip movement video clips and speech clips can be the same or different. Here, the image frames in the lip movement image sequence are images of the lip region.

[0108] In step S120, the extraction of lip movement video segments and speech segments can be performed simultaneously or in a certain order. In step S120, the VAD algorithm is used to obtain human voice interval values ​​(VAD values) while segmenting the video and audio, in order to find lip movement video segments and speech segments corresponding to the same spoken content (e.g., the same sentence or word spoken by the user), thereby determining the matching relationship between each speech segment and each lip movement video segment.

[0109] Step S130: Select the lip movement video segment that matches the speech segment.

[0110] In some embodiments, the lip movement video segments that match each speech segment can be determined based on the human voice range of the lip movement video segment and the human voice range of the speech segment, and the lip movement video segments that match each speech segment can be used as candidate lip movement video segments.

[0111] Generally, if there is a certain degree of overlap in the time dimension, the lip movement video clip and the audio clip can be considered to be synchronous, meaning that the lip movement video clip and the audio clip correspond to the same sentence or word from the same speaker. Therefore, the degree of overlap in the time dimension can be used to find the lip movement video clip that matches the audio clip.

[0112] In some embodiments, step S130 may include: determining the overlap length in the time dimension between the lip-movement video segment and the speech segment based on the voice interval (e.g., VAD value) of the lip-movement video segment and the voice interval (e.g., VAD value) of the speech segment. The lip-movement video segment and the speech segment are considered a match if the overlap length in the time dimension is greater than or equal to a preset duration threshold; otherwise, they are not a match. In other words, the speech segment and its matching lip-movement video segment satisfy the condition that the overlap length in the time dimension is greater than or equal to a preset duration threshold. This allows for efficient and accurate identification of lip-movement video segments that match each speech segment, or in other words, it allows for efficient and accurate identification of lip-reading video segments and their synchronized speech segments.

[0113] The specific value of the duration threshold can be set based on one or more factors such as the specific application scenario, the accuracy of the VAD model or algorithm, the length of the speech segment, and user needs. In some embodiments, the duration threshold can be a predetermined proportion of the speech segment length. For example, if the speech segment length is 20 seconds and the predetermined proportion is 80%, then the duration threshold is 16 seconds. In practical applications, the length of the speech segment is usually dynamically changing. Therefore, determining the duration threshold in real time by using a predetermined proportion and the speech segment length can more efficiently and accurately find the lip movement video segment that matches the speech segment, which is beneficial to improving the annotation accuracy of lip reading recognition samples.

[0114] For example, Figure 2 A schematic diagram illustrating the overlap length between a speech segment and a lip-movement video segment along the time dimension t is shown. See also Figure 2As shown, assuming the start point y1 and end point y2 of the lip movement video segment in the time dimension t are obtained through the lip movement VAD model, the human voice interval value S1 of the lip movement video segment can be represented as [y1, y2]. The start point x1 and end point x2 of the speech segment in the time dimension t are obtained through the language VAD module, and the human voice interval value S2 of the speech segment can be represented as [x1, x2]. When the overlap length S12[x1, y2] of S1 and S2 in the time dimension is greater than or equal to the preset duration threshold, the lip movement video segment matches the speech segment. That is, the lip movement video segment can be used as a candidate lip movement video segment.

[0115] Following step S130, the process may further include: selecting lip movement video segments with a lip-reading confidence level lower than a preset second confidence threshold from among the lip movement video segments that match the speech segment (i.e., from all candidate lip movement video segments obtained in step S130). The lip-reading confidence level is obtained by performing lip-reading recognition on the lip movement video segment based on a pre-obtained lip-reading recognition model. This allows the selection of the "most uncertain" or "most information-carrying" lip movement video segments as lip-reading recognition samples, effectively creating a negative sample set. This ensures that the final lip-reading recognition samples are the most difficult for the model to distinguish or the samples that provide the greatest improvement to the model, providing supervised samples for subsequent optimization of a specific lip-reading recognition model, or supervised samples for transfer learning and iterative optimization of a lip-reading recognition model for a specific person.

[0116] Similarly, after step S130, the process may further include: selecting lip movement video segments with a lip-reading confidence level greater than a preset third confidence threshold from among the lip movement video segments matching the speech segment (i.e., from all candidate lip movement video segments obtained in step S130). This allows for the selection of positive samples, forming a positive sample set, ensuring that the final lip-reading samples are conducive to training a lip-reading recognition model. This provides supervised samples for subsequent optimization of a specific lip-reading recognition model or training of a new lip-reading recognition model, or in other words, provides supervised samples for transfer learning and iterative optimization of a lip-reading recognition model for a specific individual.

[0117] Here, the confidence level of a lip-reading video clip can be obtained by performing lip-reading on the clip using parameters of a lip-reading model associated with the user's preset information. Alternatively, the confidence level can be obtained by performing lip-reading on the clip using general lip-reading model parameters.

[0118] The specific values ​​for the second and / or third confidence thresholds can be set based on one or more factors such as the specific application scenario, the accuracy of the VAD model or algorithm, the length of the speech segment, and user needs. For example, the second confidence threshold can be 0.5, 0.4, 0.3, or any other value less than 1. The third confidence threshold can be 0.5, 0.6, 0.7, or any other value less than 1.

[0119] Step S140: Label the lip-reading segment with the recognized text of the speech segment to obtain the user's lip-reading sample.

[0120] Still with Figure 2 For example, in step S140, the speech segment with the human voice interval value S2[x1,x2] is subjected to speech recognition to obtain a recognition text. This recognition text is used as a label for the lip movement video segment with the human voice interval value S1[y1,y2] to annotate the lip movement video segment, thus forming a lip movement recognition sample.

[0121] In or before step S140, a pre-obtained ASR model can be used to perform speech recognition on the speech segment, obtaining the recognized text of the speech segment, which contains the utterance content of the speech segment. In step S140, the recognized text of the speech segment is used as a label to annotate the lip-reading segment matched with the speech segment, thus obtaining the user's lip-reading sample. Here, the ASR model can be a neural network-based ASR model or any other type of ASR model.

[0122] In practical applications, speech recognition can be performed using electronic devices. Alternatively, the electronic device can upload speech segments to a cloud server, where it performs speech recognition and semantic analysis, then sends the resulting recognized text back to the electronic device, which then annotates the lip-reading segments. To protect user privacy and avoid the risks associated with data loss, it is preferable to perform speech recognition using electronic devices.

[0123] In step S140, lip movement video segments can be further filtered based on the confidence level of the speech segments. Specifically, the confidence level of the speech segment is compared with a preset third confidence threshold. If the confidence level of the speech segment is greater than or equal to the third confidence threshold, the lip movement video segment matching that speech segment is retained; if the confidence level of the speech segment is less than the third confidence threshold, the lip movement video segment matching that speech segment is discarded. In this way, lip movement video segments corresponding to speech segments with better speech quality can be selected as lip reading recognition samples, which can improve the accuracy of lip reading recognition samples.

[0124] In this embodiment, the electronic device can automatically capture the speaker's video and audio, and automatically segment the speaker's lip movement video segments using technologies such as VAD and speech recognition, labeling them as positive and negative samples. This means that speech modality can be used to assist visual modality in automatic labeling, providing lip-reading samples for supervised transfer learning and iterative optimization targeting the speaker. Furthermore, the entire process is seamless for the user, requiring no pre-labeling for specific users or asking experts to determine data labels, making it more user-friendly and effectively improving the user experience.

[0125] Figure 3 A schematic flowchart of the model training method provided in this application embodiment is shown. The model training method of this application embodiment can be executed by an electronic device; see [link to relevant documentation]. Figure 3 As shown, the model training method of this application embodiment may include the following steps S110 to S140, and step S150.

[0126] Step S150: Update the parameters of the lip reading model using the lip reading samples obtained in step S140.

[0127] The lip-reading model in this application embodiment can be any type of model. In particular, the lip-reading model can be, but is not limited to, a neural network-based lip-reading model, a sequence-to-sequence-based lip-reading model, or a connectionist temporal classification loss model. It is understood that any other lip-reading model applicable to the embodiments of this application should be included within the scope of this application, and will not be listed here.

[0128] In one implementation, step S150 can utilize the lip-reading samples obtained in step S140 to train a lip-reading model, thereby determining the parameters of the lip-reading model. In another implementation, step S150 can utilize the lip-reading samples obtained in step S140 to optimize the lip-reading model, thereby updating the parameters of the lip-reading model (e.g., the parameters of its trainable layers), achieving user customization of the lip-reading model and improving the recognition accuracy of the lip-reading model for specific users.

[0129] In step S150, gradient descent can be used to update the parameters of the lip-reading model. For example, batch gradient descent, stochastic gradient descent, or mini-batch gradient descent can be used, and the lip-reading samples obtained in step S140 can be used to update the parameters of the lip-reading model. Here, the loss function used can be the cross-entropy loss function or any other type of loss function.

[0130] Typically, a lip-reading model can include a general feature layer and a trainable layer. The parameters of the lip-reading model can include trainable layer parameters and general feature layer parameters. As the names suggest, trainable layer parameters refer to the parameters of the trainable layer in the lip-reading model, while general feature layer parameters refer to the parameters of the general feature layer in the lip-reading model. Generally, the general feature layer parameters are fixed, while the trainable layer parameters can be optimized. Therefore, updating the parameters of the lip-reading model in step S150 can specifically mean updating the trainable layer parameters of the lip-reading model. In this way, for each user, only the trainable layer parameters of the lip-reading model can be updated, which not only improves parameter optimization efficiency but also reduces the amount of data for each user's lip-reading model parameters, effectively reducing the size of the lip-reading model for each user in the electronic device and saving hardware resources.

[0131] In step S150, the lip-reading model parameters (e.g., trainable layer parameters) of the user can be obtained based on the user's preset information. These parameters are then updated using the user's lip-reading samples, resulting in lip-reading model parameters with higher accuracy for that user, thus improving the accuracy of the lip-reading model for individual users. In other words, in step S150, the parameters of the lip-reading model are those that match or are associated with the user's preset information. This continuous updating and optimization of the lip-reading model parameters using lip-reading samples from a specific user allows for dynamic adjustment of the parameters based on the user's own changes. This enables continuous learning and optimization of the lip-reading model, continuously improving its accuracy for specific users and solving the problem of the lip-reading model's inability to continuously optimize in line with user needs.

[0132] The user's preset information may be, but is not limited to, voiceprints, face IDs, lip prints, or other information, which can be obtained through corresponding modules in the electronic device. In some embodiments, the preset information may be face IDs, voiceprints, lip prints, etc. In this way, the user's preset information can be obtained directly from the video or audio in step S110, which not only reduces the amount of data, but also ensures that the preset information and the user of the lip reading sample are consistent. Such lip reading samples are more conducive to obtaining a lip reading model with high accuracy for specific users.

[0133] Step S150 may further include: adjusting the hyperparameters of the lip-reading model by comparing the lip-reading results and labels of the lip-reading samples to obtain the hyperparameters of each lip-reading sample, and then using the lip-reading samples and their corresponding hyperparameters to update the parameters of the lip-reading model. Here, the hyperparameters may include, but are not limited to, the parameter update rate (i.e., the learning rate).

[0134] The parameter update rate (i.e., the learning rate) is a crucial hyperparameter for optimizing lip-reading models. In gradient descent, the value of the parameter update rate is critical; if it's too large, convergence will be impossible, and if it's too small, convergence will be too slow. Before updating the parameters of the lip-reading model in step S150, the process may further include: adjusting the parameter update rate of the lip-reading model by comparing the lip-reading text of the lip-reading sample with the label of the lip-reading sample to obtain the parameter update rate for the corresponding lip-reading sample; wherein, the lip-reading text is obtained by the lip-reading model performing lip-reading on the lip-reading sample. Specifically, updating the parameters of the lip-reading model in step S150 involves updating the parameters of the lip-reading model using the parameter update rates of the lip-reading sample and the corresponding lip-reading sample. Specifically, when the lip reading result of a lip reading sample matches the label of the lip reading sample, the parameter update rate of the lip reading model is lowered to obtain the corresponding parameter update rate for that lip reading sample; when the lip reading result of a lip reading sample does not match the label of the lip reading sample, the parameter update rate of the lip reading model is increased to obtain the corresponding parameter update rate for that lip reading sample. This improves the optimization efficiency of the lip reading model, reduces hardware resource consumption, and further enhances the lip reading accuracy of the lip reading model for individual users.

[0135] In practical applications, the parameter update rate can be adjusted through learning rate decay, learning rate warm-up, periodic learning rate adjustment, or some adaptive learning rate adjustment methods.

[0136] In or after step S150, the method may further include: associating and saving the trainable layer parameters of the lip-reading model with the user's preset information (e.g., the face ID hereinafter). Specifically, the user's preset information can be stored in a registered information database (e.g., the registered face database hereinafter), and the trainable layer parameters associated with the preset information can be stored in a lip-reading model library. This allows for convenient and rapid retrieval of the user's trainable layer parameters using their preset information, enabling lip-reading recognition of the user and improving the accuracy of individual lip-reading. Furthermore, the existence of the user's lip-reading model parameters can be determined simply by querying the registered information database, making the process faster and more efficient.

[0137] The model training method in this application uses speech modality to assist visual modality in automatic annotation and model optimization, achieving self-supervised, personalized, and active learning of the lip-reading model. This simultaneously improves the recognition accuracy and personalization of the lip-reading model. Furthermore, the entire model training process requires no user intervention, is seamless for the user, and eliminates the need for pre-labeling of specific users, thus enhancing the user experience. In addition, the model training method in this application can be automatically completed by electronic devices. It can utilize user lip-reading samples to train the lip-reading model online, updating the model's parameters in real time. This improves lip-reading accuracy while enabling customization of the lip-reading model, and ensures that data remains within the system, effectively protecting user privacy.

[0138] Figure 4 A flowchart illustrating the lip-reading recognition method provided in this application embodiment is shown. The model training method in this application embodiment can be executed by an electronic device; see [link to relevant documentation]. Figure 4 As shown, the model training method in this application embodiment may include the following steps:

[0139] Step S410: When it is detected that the user is speaking into the electronic device, the user's video is captured.

[0140] Step S420: Extract the lip movement video segment from the video.

[0141] In some embodiments, step S420 may include: step a1, segmenting the video by lip movement detection to obtain lip movement video during the mouth movement period; step a2, performing lip movement VAD detection on the lip movement video to obtain lip movement video segments, wherein in the image sequence of each lip movement video segment, each frame image may contain only the user's lip region.

[0142] In one implementation, in step a1, the user's mouth movement period can be determined based on the real-time lip movement detection result in step S410. The lip region within the mouth movement period is extracted from the video, unit by unit, to obtain the lip movement video. Each lip movement video corresponds to one face, and each single frame in the lip movement video contains only image data of the face's lip region. Thus, the lip movement video segments obtained through the lip movement video have a small data volume, which helps reduce computational complexity and minimizes interference from other faces and their lips in the video, thereby improving the processing efficiency and accuracy of user lip reading recognition.

[0143] For example, if a video contains two people, and there is only one speaker, the resulting lip movement video will only contain the image of that speaker's lips. If there are N speakers (N is an integer greater than 1), the resulting lip movement video will be divided into N groups, each group corresponding to one speaker (for example, it can be associated with the speaker's preset information), and each lip movement video will only contain the image of the corresponding speaker's lips.

[0144] Step S430: Based on the parameters of the lip reading recognition model obtained by the above model training method, run the lip reading recognition model to perform lip reading recognition on the lip movement video clip and obtain lip reading recognition text, which can indicate the content of the user's speech.

[0145] In some embodiments, step S430 may include: obtaining the user's preset information; obtaining trainable layer parameters associated with the preset information; loading the trainable layer parameters and pre-configured general feature layer parameters to run the lip-reading recognition model to perform lip-reading recognition on the lip-reading video clip. Thus, by quickly finding the speaker's trainable layer parameters through the speaker's preset information, and using the speaker's trainable layer parameters and general feature layer parameters that can be shared by all users to run the lip-reading recognition model to perform lip-reading recognition on the speaker, it is equivalent to using the speaker's customized lip-reading recognition model to perform lip-reading recognition on the speaker, thereby effectively improving the individual-specific lip-reading recognition accuracy in a specific domain.

[0146] In practical applications, a lip-reading model library and a registered face database can be pre-created. The lip-reading model parameters (i.e., the trainable layer parameters of the lip-reading model) for each user are stored in the lip-reading model library, while pre-defined information associated with the lip-reading model parameters (e.g., face ID) is stored in the registered face database. Thus, when performing lip-reading on a specific user, the existence of the user's lip-reading model parameters can be determined by querying the registered face database to see if the user's pre-defined information exists. If it exists, the user's lip-reading model parameters can be quickly retrieved from the lip-reading model library using the user's pre-defined information. Therefore, the lip-reading model can be run using the user's lip-reading parameters to perform lip-reading on the user, which is equivalent to using the user's customized lip-reading model to perform lip-reading on the user, achieving customized lip-reading for the user and thus improving the accuracy of lip-reading for various types of people.

[0147] In some embodiments, step S430 may further include: if no lip-reading model parameters matching the preset information are found (i.e., the trainable layer parameters associated with the preset information do not exist), it indicates that the user's lip-reading model parameters do not exist. In this case, pre-configured trainable layer parameters and general feature layer parameters can be directly loaded, and the lip-reading model can be run using these parameters to perform lip-reading recognition. This is equivalent to using a general lip-reading model to perform lip-reading recognition on the user's lip-reading video clips. Thus, even when the user's lip-reading model parameters do not exist, lip-reading recognition can be performed on the user using general lip-reading model parameters, thereby efficiently completing the user's lip-reading recognition.

[0148] Here, the user's preset information can be found in the relevant description above, and will not be repeated here. In some embodiments, the preset information in step S410 may be a face ID or lip print, which can be obtained by performing face recognition on the video in step S410.

[0149] The lip reading method of this application embodiment can efficiently complete lip reading recognition for various types of people through electronic devices, and the lip reading recognition accuracy is high.

[0150] Figure 5 An exemplary structure of the lip-reading device 50 provided in this application embodiment is shown. This lip-reading device 50 can be applied to an electronic device. Specifically, the lip-reading device 50 can be installed in an electronic device or implemented directly through the software and / or hardware of the electronic device. Specific details regarding the electronic device can be found in the relevant description below, and will not be repeated here. See also Figure 5 As shown, the lip-reading device 50 may include:

[0151] The video acquisition unit 51 is configured to acquire the user's video while the user is speaking into the electronic device;

[0152] The audio acquisition unit 52 is configured to acquire the user's audio while the user is speaking into the electronic device;

[0153] Lip movement extraction unit 53 is configured to extract lip movement video segments from a video.

[0154] The speech extraction unit 54 is configured to extract speech segments from audio.

[0155] Selection unit 55 is configured to select a lip movement video segment that matches the speech segment;

[0156] Labeling unit 56 is configured to label lip movement video segments with the speech recognition text of the speech segments as labels, thereby obtaining the user's lip reading recognition samples.

[0157] In some embodiments, the lip-reading device 50 may further include one or more of the following:

[0158] Noise detection unit 57 is configured to detect the volume of ambient noise;

[0159] The wake-up voice confidence acquisition unit 58 is configured to acquire the user's wake-up voice confidence.

[0160] The face detection unit 59 is configured to detect whether a face or mouth is present within the field of view of the camera;

[0161] The positioning unit 510 is configured to acquire the speaker's position in the video and the sound source positioning direction of the audio.

[0162] Specifically, the video acquisition unit 51 can be configured to acquire the user's video when the ambient noise volume is equal to or less than a preset noise threshold, the wake-up voice confidence level is greater than or equal to a preset first confidence threshold, the camera's field of view contains a face or mouth, and / or the speaker's position in the video matches the sound source localization direction of the audio; and / or,

[0163] The audio acquisition unit 52 can be specifically configured to acquire the user's audio when the volume of the ambient noise is equal to or less than a preset noise threshold, the confidence level of the wake-up voice is greater than or equal to a preset first confidence threshold, the field of view of the camera contains a face or mouth, and / or the speaker's position in the video matches the sound source localization direction of the audio.

[0164] In some embodiments, the lip movement extraction unit 53 may be specifically configured to: perform endpoint detection and segmentation on the video using a lip movement voice interval detection method to obtain the lip movement video segment and the voice interval of the lip movement video segment; and / or, the speech extraction unit 54 may be specifically configured to: perform endpoint detection and segmentation on the audio using a speech voice interval detection method to obtain the speech segment and the voice interval of the speech segment.

[0165] In some embodiments, the selection unit 55 is specifically configured to: determine the overlap length of the speech segment and the lip-moving video segment in the time dimension based on the human voice interval of the speech segment and the human voice interval of the lip-moving video segment; when the overlap length of the speech segment and the lip-moving video segment in the time dimension is greater than or equal to a predetermined duration threshold, the speech segment and the lip-moving video segment are matched.

[0166] In some embodiments, the selection unit 55 may also be configured to: select, from the lip movement video segments that match the speech segment, the lip movement video segments whose lip reading confidence is less than a preset second confidence threshold, wherein the lip reading confidence is obtained by performing lip reading recognition on the lip movement video segments according to a pre-obtained lip reading recognition model.

[0167] In some embodiments, the lip movement video segment includes a sequence of lip movement images, wherein the image frames in the sequence are images of the lip region.

[0168] In some embodiments, the lip reading device 50 may further include a parameter update unit 511 configured to update the parameters of the lip reading model using the lip reading samples obtained by the annotation unit 56.

[0169] In some embodiments, the lip reading recognition model includes a general feature layer and a trainable layer, and the parameters of the lip reading recognition model include trainable layer parameters and general feature layer parameters; the parameter update unit 511 is specifically configured to update the trainable layer parameters of the lip reading recognition model.

[0170] In some embodiments, the lip reading device 50 may further include a storage unit 512 configured to associate and save the trainable layer parameters of the lip reading model with preset information of the user.

[0171] In some embodiments, the storage unit 512 may be specifically configured to: store the user's preset information in a registered information database; and store the trainable layer parameters associated with the preset information in a lip-reading model library.

[0172] In some embodiments, the parameter update unit 511 may be specifically configured to: adjust the parameter update rate of the lip reading model by comparing the lip reading text of the lip reading sample with the label of the lip reading sample to obtain the parameter update rate corresponding to the lip reading sample; update the parameters of the lip reading model using the lip reading sample and the parameter update rate of the corresponding lip reading sample; wherein, the lip reading text is obtained by the lip reading model performing lip reading on the lip reading sample.

[0173] In some embodiments, the video acquisition unit 51 may also be configured to acquire the user's video when the user speaks to the electronic device; the lip movement extraction unit 53 may also be configured to extract lip movement video segments from the video; the lip reading recognition device 50 may also include: a lip reading recognition unit 513, configured to run the lip reading recognition model according to the parameters of the lip reading recognition model updated by the parameter update unit 511 to perform lip reading recognition on the lip movement video segments and obtain lip reading recognition text.

[0174] In some embodiments, the lip-reading recognition model may include a general feature layer and a trainable layer, and the parameters of the lip-reading recognition model include trainable layer parameters and general feature layer parameters. The lip-reading recognition device 50 may further include: a preset information acquisition unit 514 configured to acquire preset information from the user; and a lip-reading recognition unit 513 specifically configured to: acquire trainable layer parameters associated with the preset information; load the trainable layer parameters and pre-configured general feature layer parameters to run the lip-reading recognition model to perform lip-reading recognition on lip movement video clips.

[0175] Specifically, the lip reading recognition unit 513 is configured to perform lip reading recognition on the lip movement video segment obtained by the lip movement extraction unit 53 based on the lip reading recognition model parameters trained by the parameter update unit 511, and obtain a lip reading recognition result. This lip reading recognition result may include lip reading text and lip reading recognition confidence score. For details on lip reading text and lip reading confidence score, please refer to the relevant context description, which will not be repeated here.

[0176] In this application embodiment, other technical details of the lip reading recognition device 50 can be found in the above method and the relevant descriptions of the embodiments below, and will not be repeated here.

[0177] Figure 6 Exemplary application scenarios and exemplary processing procedures of embodiments of this application are illustrated.

[0178] Figure 6 In the example, user A controls the robot's playback content via voice. User A first says "Xiao Yi Xiao Yi" containing specific command words to the robot, then says "Stop talking" to stop the robot from playing the current content, and finally says "Play music" to play music.

[0179] During the face-to-face interaction between User A and the robot, the robot captures video and audio using its camera and microphone, respectively. After VAD processing, it obtains three lip movement video clips and their VAD values, and three speech clips and their VAD values. These three lip movement video clips correspond to the three sentences mentioned above: "Hey Celia," "Don't say anything," and "Play music." The VAD values ​​of the three lip movement video clips contain the start and end points of the lip movement for each sentence. Similarly, the three speech clips also correspond to the three sentences, and their VAD values ​​contain the start and end points of the speech for each sentence. Then, the robot finds the lip movement video clip corresponding to each speech clip based on the overlap length of the VAD values, performs speech recognition on the speech clips to obtain the speech recognition text for each speech clip. The speech recognition text is the label data of the lip movement video clips corresponding to the speech clips. Using this label data to annotate the three lip movement video clips, we can obtain three lip movement recognition samples for User A: class1, class2, and class3.

[0180] like Figure 6 As shown, the label for lip movement recognition sample class 1 is the text "Xiao Yi Xiao Yi" and the lip movement video clip includes a sequence of lip region images when user A says the sentence. The label for lip movement recognition sample class 2 is the text "Don't say it" and the lip movement video clip includes a sequence of lip region images when user A says the sentence. The label for lip movement recognition sample class 3 is the text "Play music" and the video clip includes a sequence of lip region images when user A says the sentence.

[0181] The robot uses three lip movement recognition samples (class1, class2, and class3) from user A to update the trainable layer parameters of user A or general trainable layer parameters. These updated trainable layer parameters are then associated with user A's face ID and stored in the robot's lip-reading model library. This achieves online optimization and active learning of the lip-reading model, improving its accuracy in recognizing user A's lip movements. In other words, it obtains customized lip-reading model parameters for user A, realizing user customization of the lip-reading model.

[0182] Specifically, the robot can perform lip movement recognition and face recognition on the video to obtain user A's facial feature data. It then uses this data to query the registered face database to see if a corresponding face ID exists. If the face ID exists, the robot uses it to query the lip-reading model library for user A's trainable layer parameters. The robot updates these parameters using three lip movement recognition samples (class1, class2, class3) and stores the updated parameters in the lip-reading model library. If the face ID does not exist, the robot updates the locally stored trainable layer parameters using the three samples, configuring the corresponding face ID from the facial feature data to obtain the user A's face ID. This face ID is then stored in the registered face database. Finally, the updated trainable layer parameters are associated with the face ID and stored in the lip-reading model library.

[0183] After the above process, when User A speaks to the robot again, the robot can obtain User A's face ID by performing facial recognition on User A, obtain User A's trainable layer parameters through User A's face ID, load User A's trainable layer parameters and pre-stored general feature layer parameters locally to run the lip-reading recognition model to perform lip-reading recognition on User A. This is equivalent to using User A's customized lip-reading recognition model to perform real-time lip-reading recognition on User A, thereby effectively improving User A's lip-reading recognition accuracy.

[0184] Figure 7 This document illustrates an exemplary implementation process for lip reading recognition, sample annotation, and model training in embodiments of this application.

[0185] See Figure 7 As shown, the process of lip reading recognition using a lip reading model can include the following steps:

[0186] Step S711, lip movement detection, that is, detecting whether the speaker is speaking at each moment.

[0187] Step S712, video segmentation to obtain the speaker's lip movement video segment, that is, based on the lip movement detection results, extract the video sequence of the lip region within the mouth movement time period (e.g., 60 frames of 112*112 image data) (i.e., lip movement video) for each face region as a unit.

[0188] Step S713, face recognition, that is, performing face recognition on the speaker's lip movement video clip to obtain the speaker's (e.g., Figure 6 The facial feature data of user A in the data.

[0189] Here, facial feature data can be, but is not limited to, facial landmark data, mouth landmark data, or others. In practical applications, existing facial landmark detection algorithms can be used to perform face recognition on image frames in a video to obtain facial feature data.

[0190] Step S714: Use the speaker's facial feature data to query the registered face database for the speaker's face ID. If the speaker's face ID is matched, it means that the speaker's own lip reading model parameters already exist, and you can continue to step S715. If the speaker's face ID is not matched, it means that the speaker's lip reading model parameters do not yet exist, and you can jump to step S716.

[0191] The registered face database contains all face IDs associated with lip-reading model parameters (or the parameters of the trainable layers of the lip-reading model). If a face ID is matched in the registered face database, it indicates that the speaker's lip-reading model parameters (e.g., the trainable layers of the lip-reading model) already exist, which is equivalent to the speaker's customized lip-reading model already existing. If a face ID is not matched in the registered face database, it indicates that the speaker's lip-reading model parameters do not yet exist, which is equivalent to the speaker's customized lip-reading model not existing.

[0192] Step S715: Load the lip reading model parameters of the corresponding speaker's face ID into memory, and run the lip reading model using these parameters;

[0193] The lip-reading model parameters corresponding to the speaker's face ID include trainable layer parameters of the speaker's face ID and locally stored general feature layer parameters. The speaker's face ID is used to retrieve the speaker's trainable layer parameters from the lip-reading model library. These trainable layer parameters, along with the locally stored general feature layer parameters (i.e., general feature layer parameters applicable to all types of users, or those shared by all types of users), are loaded into memory. The electronic device's processor uses these parameters to run the lip-reading model for lip-reading recognition. This is equivalent to using a user-customized lip-reading model to perform lip-reading on the user's lip-reading video clips. Using this customized lip-reading model for specific users results in higher accuracy and better performance. It should be noted that this customized lip-reading model can not only perform lip-reading for specific users but also for other types of users.

[0194] Step S716: Load the general lip reading model parameters into memory and run the lip reading model using these parameters;

[0195] A general lip-reading model includes locally stored trainable layer parameters (i.e., trainable layer parameters applicable to all types of users) and locally stored general feature layer parameters (i.e., general feature layer parameters applicable to all types of users, or general feature layer parameters shared by all types of users). The locally stored trainable layer parameters and the locally stored general feature layer parameters are loaded into memory. The processor in the electronic device uses these parameters to run the lip-reading model for lip-reading. This is equivalent to using a general lip-reading model to perform lip-reading on a user's lip-reading video clip. This general lip-reading model can be applicable to all types of users, enabling lip-reading for all users. However, the accuracy of this general lip-reading model may be lower for certain users or a particular user (or most users).

[0196] Step S717: Perform lip reading recognition on the lip reading video segment obtained in step S712 and output the result. The result may include lip reading recognition text and lip reading recognition confidence score. The lip reading recognition confidence score can indicate the credibility of the lip reading recognition text obtained by performing lip reading recognition on the lip reading video segment.

[0197] It should be noted that the general lip reading model and the customized lip reading model for each user are essentially the same lip reading model. They have the same general feature layer parameters, but different trainable layer parameters.

[0198] See Figure 7 As shown, sample labeling and model training can be performed in a quiet environment, and the specific implementation process can include the following steps:

[0199] Step S721, Voice wake-up successful: The speaker speaks a wake-up voice containing a specific command word (e.g., "Xiaoyi Xiaoyi") to the electronic device, successfully waking up the electronic device.

[0200] Step S722, Environmental noise perception: Perceive surrounding environmental noise. If the noise is below the noise threshold and the wake-up confidence is above the first confidence threshold, continue to step S723; otherwise, exit the current process.

[0201] Step S723, Sample validity judgment: Detect whether the face is within the field of view and whether the sound source localization direction is consistent with the face position. If the face is within the field of view and the sound source localization direction is consistent with the face position, the sample is valid, and continue to step S724; if the sound source localization direction is inconsistent with the face position and / or the face is not within the field of view, the sample is invalid, and exit the current process.

[0202] In step S724, the speaker speaks into the electronic device, the camera captures the speaker's video, and the microphone captures the speaker's audio. The lip movement VAD model extracts lip movement video segments from the video, and the speech VAD model extracts speech segments from the audio. Based on the VAD values ​​of the lip movement video segments and the speech segments, the corresponding lip movement video segments for each speech segment are determined. The speech recognition text of the speech segments is used as a label to annotate the corresponding lip movement video segments, thereby obtaining the speaker's lip reading recognition samples.

[0203] Specifically, the speaker's video is segmented and endpoints are detected using a lip movement VAD model to obtain lip movement video segments and their VAD values; and the speaker's audio is segmented and endpoints are detected and audio segments using a speech VAD model to obtain speech segments and their VAD values. Lip movement video segments with an overlap length greater than a duration threshold in the time dimension are identified as lip movement video segments corresponding to speech segments. The lip movement video segments corresponding to each speech segment are used as candidate lip movement video segments for lip reading recognition samples. Speech recognition is performed on each speech segment to obtain its speech recognition text. The corresponding candidate lip movement video segments are labeled with the speech recognition text of the speech segments to obtain the speaker's lip reading recognition samples.

[0204] Step S725: Perform lip reading recognition on the lip reading recognition samples (i.e., the lip movement video clips selected in step S724) to obtain the recognition results of each lip reading recognition sample. The recognition results may include lip movement recognition text and lip reading recognition confidence.

[0205] Here, the lip reading recognition model can be run in the manner of steps S713 to S715 to perform lip reading recognition on the lip reading samples in order to obtain the recognition results of the lip reading samples.

[0206] Step S726: Based on the recognition results obtained in step S725 and the labels marked in step S724, set the parameter update rate for the corresponding lip reading recognition sample.

[0207] Specifically, the lip-reading text of the lip-movement video clip is compared with the speech-reading text of the corresponding speech clip. If they match, the lip-reading is considered correct, and the parameter update rate of the lip-reading model can be lowered to obtain the parameter update rate of the corresponding lip-reading sample. If they do not match or the difference is large, it indicates that the lip-reading is incorrect, and the parameter update rate of the lip-reading model can be increased to obtain the parameter update rate of the corresponding lip-reading sample.

[0208] In step S727, the parameters of the lip-reading model are updated using the lip-reading samples and the parameter update rate of the lip-reading samples obtained in step S724 (e.g., updating the trainable layer parameters associated with the speaker's face ID or general trainable layer parameters). The updated lip-reading model parameters (e.g., the trainable layer parameters associated with the speaker's face ID) are associated with the speaker's face ID and updated in the lip-reading model library. The speaker's face ID corresponds to the speaker's facial feature data and the speaker's face ID is stored in the registered face database. Thus, the speaker's lip-reading model parameters can be obtained.

[0209] pass Figure 6 and Figure 7 As the examples demonstrate, model training and optimization, as well as lip reading, can all be completed by electronic devices. This ensures that legitimate and compliant users update the model locally, preventing data from being exported (i.e., no need to upload to the cloud), thus effectively protecting user privacy. Simultaneously, when a speaker engages in face-to-face communication with an electronic device (such as a robot), the accuracy of lip reading can be continuously and effectively improved for different speakers. Furthermore, in noisy environments, electronic devices can improve their wake-up rate by performing high-accuracy lip reading of the speaker.

[0210] Figure 8 This is a schematic structural diagram of an electronic device 80 provided in an embodiment of this application. The electronic device 80 includes a processor 81 and a memory 82. When the electronic device 80 is running, the processor 81 executes computer execution instructions stored in the memory 82 to perform the steps of the above-described sample annotation method, model training method, and / or lip reading recognition method.

[0211] Processor 81 can be connected to memory 82. Memory 82 can be used to store program code and data. Therefore, memory 102 can be a storage unit inside processor 101, an external storage unit independent of processor 81, or a component that includes both internal storage units of processor 81 and external storage units independent of processor 81.

[0212] It should be understood that in the embodiments of this application, the processor 81 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 81 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0213] Memory 82 may include read-only memory and random access memory, and provides instructions and data to processor 81. A portion of processor 81 may also include non-volatile random access memory. For example, processor 81 may also store device type information.

[0214] Electronic device 80 may also include a communication interface 83. It should be understood that... Figure 8 The communication interface 83 in the electronic device 80 shown can be used to communicate with other devices.

[0215] Optionally, the electronic device 80 may also include a bus. The memory 82 and communication interface 83 can be connected to the processor 81 via the bus. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0216] It should be understood that the electronic device 80 according to the embodiments of this application can correspond to the corresponding subject executing the methods according to the various embodiments of this application, and the above and other operations and / or functions of each module in the electronic device 80 are respectively for implementing the corresponding processes of the methods of this embodiment. For the sake of brevity, they will not be described in detail here.

[0217] The electronic device 80 can be, but is not limited to, at least one of the following: mobile phone, foldable electronic device, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device. This application embodiment does not impose any special limitation on the specific type of the electronic device 80.

[0218] Figure 9 This illustration shows an exemplary specific implementation structure of the electronic device 80 according to an embodiment of this application, namely the electronic device 90.

[0219] Electronic device 90 may include processor 910, external memory interface 920, internal memory 921, universal serial bus (USB) connector 930, charging management module 940, power management module 941, battery 942, antenna 1, antenna 2, mobile communication module 950, wireless communication module 960, audio module 970, speaker 970A, receiver 970B, microphone 970C, headphone jack 970D, sensor module 980, button 990, motor 991, indicator 992, camera 993, display screen 994, and subscriber identification module (SIM) card interface 995, etc. The sensor module 980 may include a pressure sensor 980A, a gyroscope sensor 980B, a barometric pressure sensor 980C, a magnetic sensor 980D, an accelerometer sensor 980E, a distance sensor 980F, a proximity sensor 980G, a fingerprint sensor 980H, a temperature sensor 980J, a touch sensor 980K, an ambient light sensor 980L, a bone conduction sensor 980M, etc.

[0220] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 90. In other embodiments of this application, the electronic device 90 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0221] Processor 910 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0222] The processor 910 can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0223] The processor 910 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 910 may be a cache memory. This memory can store instructions or data that the processor 910 has used or that are used frequently. If the processor 910 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 910, and thus improves the efficiency of the system.

[0224] In some embodiments, the processor 910 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 910 can connect to modules such as touch sensors, audio modules, wireless communication modules, displays, and cameras through at least one of these interfaces.

[0225] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 90. In other embodiments of this application, the electronic device 90 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0226] The USB connector 930 is a USB standard-compliant interface used to connect electronic device 90 to peripheral devices, specifically Mini USB, Micro USB, USB Type-C, etc. The USB connector 930 can be used to connect a charger to charge the electronic device 90, or to connect other electronic devices to enable data transfer between them. It can also be used to connect headphones to output audio stored on the electronic device. Furthermore, the connector can be used to connect other electronic devices, such as VR devices. In some embodiments, the Universal Serial Bus standard specification can be USB 1.x, USB 2.0, USB 3.x, or USB 4.

[0227] The charging management module 940 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 940 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 940 receives wireless charging input via the wireless charging coil of the electronic device 90. While charging the battery 942, the charging management module 940 can also supply power to the electronic device via the power management module 941.

[0228] The power management module 941 connects the battery 942, the charging management module 940, and the processor 910. The power management module 941 receives input from the battery 942 and / or the charging management module 940, providing power to the processor 910, internal memory 921, display screen 994, camera 993, and wireless communication module 960. The power management module 941 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 941 may also be located within the processor 910. In other embodiments, the power management module 941 and the charging management module 940 may be located in the same device.

[0229] The wireless communication function of electronic device 90 can be realized through antenna 1, antenna 2, mobile communication module 950, wireless communication module 960, modem processor and baseband processor, etc.

[0230] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 90 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0231] The mobile communication module 950 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 90. The mobile communication module 950 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 950 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 950 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 950 may be housed in the processor 910. In some embodiments, at least some functional modules of the mobile communication module 950 and at least some modules of the processor 910 may be housed in the same device.

[0232] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to a speaker 970A, receiver 970B, etc.) or displays images or videos through a display screen 994. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 910 and may be housed in the same device as the mobile communication module 950 or other functional modules.

[0233] The wireless communication module 960 can provide solutions for wireless communication applications on the electronic device 90, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), ultra-wideband (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 960 can be one or more devices integrating at least one communication processing module. The wireless communication module 960 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 910. The wireless communication module 960 can also receive signals to be transmitted from processor 910, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0234] In some embodiments, antenna 1 of electronic device 90 is coupled to mobile communication module 950, and antenna 2 is coupled to wireless communication module 960, enabling electronic device 90 to communicate with networks and other electronic devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0235] Electronic device 90 can implement display functions through a GPU, a display screen 994, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 994 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 910 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0236] Display screen 994 is used to display images, videos, etc. Display screen 994 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 90 may include one or more display screens 994.

[0237] Electronic device 90 can realize camera function through camera 993, ISP, video codec, GPU, display 994, AP, NPU, etc.

[0238] Camera 993 can be used to acquire color image data and depth data of the subject. An ISP (Image Signal Processor) can be used to process the color image data acquired by camera 993. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then transmitted to the ISP for processing, transforming it into a visible image. The ISP can also perform algorithmic optimizations on image noise, brightness, etc. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into camera 993.

[0239] In some embodiments, the camera 993 may include a color camera module and a 3D sensing module.

[0240] In some embodiments, the photosensitive element of the color camera module can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP (Image Signal Processor) for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats.

[0241] In some embodiments, the 3D sensing module can be a time-of-flight (TOF) 3D sensing module or a structured light 3D sensing module. Structured light 3D sensing is an active depth sensing technology, and its basic components may include an infrared emitter, an IR camera module, etc. Similarly, TOF 3D sensing can be an active depth sensing technology, and its basic components may include an infrared emitter, an IR camera module, etc. Structured light 3D sensing modules can also be applied to facial recognition, motion-sensing game consoles, and industrial machine vision inspection. TOF 3D sensing modules can also be applied to game consoles, augmented reality (AR) / virtual reality (VR) fields.

[0242] In other embodiments, camera 993 may also consist of two or more cameras. These two or more cameras may include a color camera, which can be used to acquire color image data of the object being photographed.

[0243] In some embodiments, the electronic device 90 may include one or more cameras 993. Specifically, the electronic device 90 may include one front-facing camera 993 and one rear-facing camera 993. The front-facing camera 993 is typically used to capture color image data and depth data of the photographer facing the display screen 994, while the rear-facing camera module is used to capture color image data and depth data of the subject (such as a person, landscape, etc.) in front of the photographer.

[0244] Digital signal processors (DSPs) are used to process digital signals, and can also process other digital signals. For example, when electronic device 90 is selecting a frequency point, the DSP is used to perform Fourier transforms on the frequency energy, etc.

[0245] Video codecs are used to compress or decompress digital video. Electronic device 90 can support one or more video codecs. Thus, electronic device 90 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0246] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable applications such as intelligent cognition in electronic devices, including image recognition, facial recognition, speech recognition, text understanding, and lip reading.

[0247] The external storage interface 920 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 90. The external memory card communicates with the processor 910 through the external storage interface 920 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card, or music, video, and other files can be transferred from the electronic device to the external memory card.

[0248] Internal memory 921 can be used to store computer executable program code, including instructions. Internal memory 921 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 90 (such as audio data, phonebook, etc.). Furthermore, internal memory 921 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 910 executes various functional methods or data processing of electronic device 90 by running instructions stored in internal memory 921 and / or instructions stored in memory disposed in the processor.

[0249] Electronic device 90 can implement audio functions such as music playback and recording through audio module 970, speaker 970A, receiver 970B, microphone 970C, headphone jack 970D, and application processor.

[0250] The audio module 970 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 970 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 970 may be located in the processor 910, or some functional modules of the audio module 970 may be located in the processor 910.

[0251] The speaker 970A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 90 can listen to music through the speaker 970A or output audio signals for hands-free calling.

[0252] The receiver 970B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 90 answers a telephone call or voice message, the receiver 970B can be brought close to the listener's ear to hear the voice.

[0253] Microphone 970C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 970C, inputting the sound signal into microphone 970C. Electronic device 90 may have at least one microphone 970C. In some embodiments, electronic device 90 may have two microphones 970C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 90 may also have three, four, or more microphones 970C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0254] The 970D headphone jack is used to connect wired headphones. The 970D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0255] Pressure sensor 980A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 980A can be disposed on display screen 994. There are many types of pressure sensors 980A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 980A, the capacitance between the electrodes changes. Electronic device 90 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 994, electronic device 90 detects the intensity of the touch operation based on pressure sensor 980A. Electronic device 90 can also calculate the touch position based on the detection signal from pressure sensor 980A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0256] The gyroscope sensor 980B can be used to determine the motion attitude of the electronic device 90. In some embodiments, the gyroscope sensor 980B can determine the angular velocity of the electronic device 90 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 980B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 980B detects the angle of the shake of the electronic device 90, calculates the distance that the lens module needs to compensate based on the angle, and controls the lens to move in the opposite direction to counteract the shake of the electronic device 90, thus achieving image stabilization. The gyroscope sensor 980B can also be used in navigation and motion-sensing game scenarios.

[0257] The barometric pressure sensor 980C is used to measure air pressure. In some embodiments, the electronic device 90 calculates altitude based on the air pressure value measured by the barometric pressure sensor 980C to assist in positioning and navigation.

[0258] The magnetic sensor 980D includes a Hall effect sensor. The electronic device 90 can use the magnetic sensor 980D to detect the opening and closing of the flip cover. When the electronic device is a foldable device, the magnetic sensor 980D can be used to detect the folding or unfolding of the electronic device, or the folding angle. In some embodiments, when the electronic device 90 is a flip phone, the electronic device 90 can detect the opening and closing of the flip cover based on the magnetic sensor 980D. Furthermore, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0259] The 980E accelerometer sensor can detect the magnitude of acceleration in various directions (typically three axes) of an electronic device 90. When the electronic device 90 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applicable to screen orientation switching, pedometers, and other applications.

[0260] A distance sensor 980F is used to measure distance. Electronic device 90 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 90 can utilize the distance sensor 980F to measure distance for rapid focusing.

[0261] The proximity sensor 980G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 90 emits infrared light outward through the LED. The electronic device 90 uses the photodiode to detect infrared reflected light from nearby objects. When the intensity of the detected reflected light is greater than a threshold, it can be determined that there is an object near the electronic device 90. When the intensity of the detected reflected light is less than the threshold, the electronic device 90 can determine that there is no object near it. The electronic device 90 can use the proximity sensor 980G to detect when a user holds the electronic device 90 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 980G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0262] The ambient light sensor 980L can be used to sense ambient light intensity. Electronic device 90 can adaptively adjust the brightness of its display screen 994 based on the sensed ambient light intensity. The ambient light sensor 980L can also be used to automatically adjust white balance when taking photos. The ambient light sensor 980L can also work in conjunction with the proximity sensor 980G to detect whether electronic device 90 is obstructed, such as when the electronic device is in a pocket. When obstruction or being in a pocket is detected, certain functions (such as touch functionality) can be disabled to prevent accidental operation.

[0263] The fingerprint sensor 980H is used to collect fingerprints. The electronic device 90 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0264] Temperature sensor 980J is used to detect temperature. In some embodiments, electronic device 90 uses the temperature detected by temperature sensor 980J to execute a temperature handling strategy. For example, when the temperature detected by temperature sensor 980J exceeds a threshold, electronic device 90 reduces processor performance to reduce power consumption and implement thermal protection. In other embodiments, when the temperature detected by temperature sensor 980J is below another threshold, electronic device 90 heats battery 942. In still other embodiments, when the temperature is below yet another threshold, electronic device 90 may boost the output voltage of battery 942.

[0265] Touch sensor 980K, also known as a "touch device," can be located on display screen 994. The touch sensor 980K and display screen 994 together form a touchscreen, also known as a "touchscreen." Touch sensor 980K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 994. In some embodiments, touch sensor 980K may also be located on the surface of electronic device 90, in a different position than display screen 994.

[0266] The bone conduction sensor 980M can acquire vibration signals. In some embodiments, the bone conduction sensor 980M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 980M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 980M can also be integrated into headphones to form bone conduction headphones. The audio module 970 can parse the voice signals based on the vibration signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 980M to realize voice functionality. The application processor can parse heart rate information based on the blood pressure signals acquired by the bone conduction sensor 980M to realize heart rate detection functionality.

[0267] Button 990 may include a power button, volume buttons, etc. Button 990 may be a mechanical button or a touch button. Electronic device 90 can receive button input and generate key signal inputs related to user settings and function control of electronic device 90.

[0268] Motor 991 can generate vibration alerts. Motor 991 can be used for incoming call vibration alerts and for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 991 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 994. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0269] Indicator 992 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0270] The SIM card interface 995 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 995 to make contact with and separate from the electronic device 90. The electronic device 90 can support one or more SIM card interfaces. The SIM card interface 995 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 995 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 995 is also compatible with different types of SIM cards. The SIM card interface 995 is also compatible with external memory cards. The electronic device 90 interacts with the network through the SIM card to achieve functions such as calls and data communication. In some embodiments, the electronic device 90 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 90 and cannot be separated from the electronic device 90.

[0271] The software system of the electronic device 90 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software structure of the electronic device 90.

[0272] Figure 10 This is an exemplary software structure block diagram of an electronic device 90 according to an embodiment of this application.

[0273] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime (ART) and native C / C++ libraries, the Hardware Abstraction Layer (HAL), and the kernel layer.

[0274] The application layer can include a series of application packages.

[0275] like Figure 10 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0276] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0277] like Figure 10 As shown, the application framework layer may include a window manager, content provider, view system, resource manager, notification manager, activity manager, input manager, etc.

[0278] The window manager provides Window Manager Service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for the input system.

[0279] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.

[0280] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0281] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0282] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0283] The Activity Manager Service (AMS) can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers), as well as manage and schedule application processes.

[0284] The Input Manager Service (IMS) provides input management services, which can be used to manage system inputs such as touchscreen input, keypad input, and sensor input. IMS retrieves events from input device nodes and, through interaction with the WMS (Windows Management System), distributes these events to appropriate windows.

[0285] The Android runtime consists of the core libraries and the Android runtime itself. The Android runtime is responsible for converting source code into machine code. The Android runtime primarily employs ahead-of-time (AOT) compilation and just-in-time (JIT) compilation techniques.

[0286] The core library primarily provides basic Java class library functionalities, such as libraries for fundamental data structures, mathematics, I / O, tools, databases, and networking. It also provides APIs for users to develop Android applications.

[0287] Native C / C++ libraries can include multiple functional modules. Examples include: surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0288] The Surface Manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The Media Framework supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. OpenGL ES provides drawing and manipulation of 2D and 3D graphics in applications. SQLite provides a lightweight relational database for applications on electronic devices.

[0289] The Hardware Abstraction Layer (HAL) runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0290] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0291] The following example, using a scene of capturing a photograph, illustrates the workflow of the electronic device 90's software and hardware.

[0292] When the touch sensor 980K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a single touch operation as an example, where the corresponding control is the camera application icon, the camera application calls the interface of the application framework layer to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through the camera 993.

[0293] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0294] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0295] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0296] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0297] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0298] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0299] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to perform the sample annotation method, model training method, and / or lip reading method described in the above embodiments.

[0300] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0301] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0302] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0303] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0304] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, all of which fall within the scope of protection of this application.

Claims

1. A sample labeling method, characterized by, The sample labeling method is applied to an electronic device, and the sample labeling method comprises: acquiring a video and audio of a user in a process in which the user speaks to the electronic device; wherein the step of acquiring the video and audio of the user is performed when a volume of environmental noise is equal to or less than a preset noise threshold and / or a speaker position of the video and a sound source positioning direction of the audio match; extracting a lip movement video segment in the video and a speech segment in the audio; selecting the lip movement video segment matching the speech segment; labeling the lip movement video segment with speech recognition text of the speech segment as a label to obtain a lip reading sample of the user; The selecting the lip movement video segment matching the speech segment comprises: determining an overlapping length of the speech segment and the lip movement video segment in a time dimension according to a vocal interval of the speech segment and a vocal interval of the lip movement video segment; the speech segment and the lip movement video segment are matched when the overlapping length of the speech segment and the lip movement video segment in the time dimension is greater than or equal to a predetermined time threshold.

2. The method of claim 1, wherein the method further comprises one or more of: obtaining a wake-up speech confidence of the user; detecting whether a field of view of a camera contains a face or a mouth; the acquiring the video and audio of the user specifically comprises: acquiring the video and audio of the user when the wake-up speech confidence is greater than or equal to a preset first confidence threshold and / or the field of view of the camera contains the face or the mouth. The extracting the lip movement video segment in the video and the speech segment in the audio comprises:

3. The method according to claim 1 or 2, characterized in that, performing endpoint detection and segmentation on the video by a lip movement vocal interval detection method to obtain the lip movement video segment and a vocal interval of the lip movement video segment; and / or, performing endpoint detection and segmentation on the audio by a speech vocal interval detection method to obtain the speech segment and a vocal interval of the speech segment. The method further comprises: in the lip movement video segment matching the speech segment, selecting the lip movement video segment with a lip reading confidence less than a preset second confidence threshold, the lip reading confidence being obtained according to lip reading of the lip movement video segment by a pre-obtained lip reading model.

4. The method according to any one of claims 1 to 2, characterized in that, The method further comprises: in the lip movement video segment matching the speech segment, selecting the lip movement video segment with a lip reading confidence less than a preset second confidence threshold, the lip reading confidence being obtained according to lip reading of the lip movement video segment by a pre-obtained lip reading model.

5. The method of claim 3, wherein, The method is applied to an electronic device, and comprises: updating parameters of a lip reading model by using the lip reading sample obtained by the sample labeling method of any one of claims 1 to 5.

6. A model training method, comprising:

7. The method of claim 6, wherein the lip reading model comprises a general feature layer and a trainable layer, and the parameters of the lip reading model comprise trainable layer parameters and general feature layer parameters; and the updating the parameters of the lip reading model specifically comprises: updating the trainable layer parameters of the lip reading model. Further comprising: ​ ​ 8. The method according to claim 6 or 7, characterized in that, ​ store preset information of a user in a registered information database; store trainable layer parameters associated with the preset information in a lip-reading model library.

9. The method of claim 6 or 7, further comprising: before the updating the parameters of the lip-reading model, adjusting an update rate of the parameters of the lip-reading model by comparing a lip-reading text of the lip-reading sample and a label of the lip-reading sample to obtain the update rate of the parameters corresponding to the lip-reading sample, wherein the lip-reading text is obtained by lip-reading the lip-reading sample using the lip-reading model; the updating the parameters of the lip-reading model comprises updating the parameters of the lip-reading model using the lip-reading sample and the update rate of the parameters corresponding to the lip-reading sample.

10. The method of claim 8, further comprising: before the updating the parameters of the lip-reading model, adjusting an update rate of the parameters of the lip-reading model by comparing a lip-reading text of the lip-reading sample and a label of the lip-reading sample to obtain the update rate of the parameters corresponding to the lip-reading sample, wherein the lip-reading text is obtained by lip-reading the lip-reading sample using the lip-reading model; the updating the parameters of the lip-reading model comprises updating the parameters of the lip-reading model using the lip-reading sample and the update rate of the parameters corresponding to the lip-reading sample.

11. A lip reading method, characterized by, applicable to an electronic device, comprising: when detecting that a user is speaking towards the electronic device, obtaining a video of the user; extracting a lip movement video segment from the video; running the lip-reading model based on the parameters of the lip-reading model obtained using the model training method of any one of claims 6 to 10 to perform lip-reading on the lip movement video segment to obtain a lip-reading text.

12. The method of claim 11, wherein: the lip-reading model comprises a general feature layer and a trainable layer, and the parameters of the lip-reading model comprise trainable layer parameters and general feature layer parameters; running the lip-reading model based on the obtained parameters of the lip-reading model to perform lip-reading on the lip movement video segment comprises: obtaining preset information of the user; obtaining trainable layer parameters associated with the preset information; loading the trainable layer parameters and pre-configured general feature layer parameters to run the lip-reading model to perform lip-reading on the lip movement video segment.

13. The method of claim 12, wherein: the preset information comprises a face ID; the obtaining of the preset information of the user comprises performing face recognition on the video to obtain face feature data of the user, and querying a registered face database for a face ID corresponding to the face feature data.

14. A lip reading apparatus characterized by comprising: The lip speech recognition device is applied to an electronic device, and the lip speech recognition device comprises: a noise detection unit configured to detect a volume of environmental noise; a positioning unit configured to obtain a speaker position in a video and a sound source positioning direction of audio; a video acquisition unit configured to acquire a video of a user in a process in which the user speaks to the electronic device; wherein the video acquisition unit is specifically configured to acquire the video of the user when the volume of the environmental noise is equal to or less than a preset noise threshold and / or the speaker position of the video matches the sound source positioning direction of the audio; an audio acquisition unit configured to acquire audio of a user in a process in which the user speaks to the electronic device; wherein the audio acquisition unit is specifically configured to acquire the audio of the user when the volume of the environmental noise is equal to or less than a preset noise threshold and / or the speaker position of the video matches the sound source positioning direction of the audio; a lip movement extraction unit configured to extract a lip movement video segment in the video; a speech extraction unit configured to extract a speech segment in the audio; a selection unit configured to select the lip movement video segment that matches the speech segment; a labeling unit configured to label the lip movement video segment by taking speech recognition text of the speech segment as a label to obtain a lip speech recognition sample of the user; the selection unit is specifically configured to: determine an overlapping length of the speech segment and the lip movement video segment in a time dimension according to a vocal interval of the speech segment and a vocal interval of the lip movement video segment; the speech segment and the lip movement video segment are matched when the overlapping length of the speech segment and the lip movement video segment in the time dimension is greater than or equal to a predetermined time threshold.

15. The device of claim 14, wherein: the lip speech recognition device further comprises one or more of the following: a wake-up speech confidence acquisition unit configured to acquire a wake-up speech confidence of the user; a face detection unit configured to detect whether a face or a mouth is contained in a field of view of a camera; the video acquisition unit is specifically configured to: the wake-up speech confidence is greater than or equal to a preset first confidence threshold, and / or the face or the mouth is contained in the field of view of the camera; and / or the audio acquisition unit is specifically configured to: the wake-up speech confidence is greater than or equal to a preset first confidence threshold, and / or the face or the mouth is contained in the field of view of the camera.

16. The device of claim 14 or 15, wherein: the lip movement extraction unit is specifically configured to perform endpoint detection and segmentation on the video by a lip movement vocal interval detection method to obtain the lip movement video segment and the vocal interval of the lip movement video segment; and / or the speech extraction unit is specifically configured to perform endpoint detection and segmentation on the audio by a speech vocal interval detection method to obtain the speech segment and the vocal interval of the speech segment.

17. The apparatus of any one of claims 14-15, wherein, The selection unit is further configured to select, from the lip movement video segments matching the voice segment, the lip movement video segment with a lip reading confidence less than a preset second confidence threshold, the lip reading confidence being obtained by performing lip reading on the lip movement video segment according to a pre-obtained lip reading model.

18. The apparatus of claim 16, wherein, The selection unit is further configured to select, from the lip movement video segments matching the voice segment, the lip movement video segment with a lip reading confidence less than a preset second confidence threshold, the lip reading confidence being obtained by performing lip reading on the lip movement video segment according to a pre-obtained lip reading model.

19. The apparatus of any one of claims 14-15, wherein, Further comprising: The parameter updating unit is configured to update parameters of the lip reading model by using the lip reading sample obtained by the labeling unit.

20. The apparatus of claim 16, wherein, Further comprising: The parameter updating unit is configured to update parameters of the lip reading model by using the lip reading sample obtained by the labeling unit.

21. The apparatus of claim 17, wherein, Further comprising: The parameter updating unit is configured to update parameters of the lip reading model by using the lip reading sample obtained by the labeling unit.

22. The apparatus of claim 19, wherein The lip reading model comprises a general feature layer and a trainable layer, and the parameters of the lip reading model comprise trainable layer parameters and general feature layer parameters.

23. The apparatus of claim 19, wherein, The parameter updating unit is specifically configured to update the trainable layer parameters of the lip reading model. Further comprising: The storage unit is configured to store preset information of a user in a registered information database; 24. The apparatus of claim 22, wherein, and store the trainable layer parameters associated with the preset information in a lip model library. Further comprising: The storage unit is configured to store preset information of a user in a registered information database; 25. The apparatus of claim 19, wherein, and store the trainable layer parameters associated with the preset information in a lip model library. The parameter updating unit is specifically configured to: adjust an update rate of the parameters of the lip reading model by comparing a lip reading text of the lip reading sample and a label of the lip reading sample, to obtain the update rate of the parameters corresponding to the lip reading sample; update the parameters of the lip reading model by using the lip reading sample and the update rate of the parameters corresponding to the lip reading sample; 26. The apparatus of any one of claims 20-22, wherein, wherein the lip reading text is obtained by performing lip reading on the lip reading sample by using the lip reading model. The parameter updating unit is specifically configured to: adjust an update rate of the parameters of the lips reading model by comparing a lip reading text of the lip reading sample and a label of the lip sample, to obtain the update rate of the parameters corresponding to the lip reading sample; update the parameter of the lip reading model by using the lip reading sample and the update rate of the parameters corresponding the lip reading sample; 27. The apparatus of claim 23, wherein, wherein the lip reading text is obtained by performing lip reading on the lips reading sample by using the lip reading model. The parameter updating unit is specifically configured to: adjust the update rate of the parameters of the lip reading model by comparing the lip reading text of the lip reading sample and the label of the lip reading sample, to obtain the update rate of the parameters corresponding to the lip sample; The parameter updating unit is specifically configured to: The parameter updating unit is specifically configured to:

28. The apparatus of claim 24, wherein, The parameter updating unit is specifically configured to: The parameter updating unit is specifically configured to: The parameter updating unit is specifically configured to: The parameter updating unit is specifically configured to: The parameter updating unit is specifically configured to:

29. The apparatus of any of claims 14-15, 18, 20-22, 25, 27, 28, wherein: The video acquisition unit is further configured to acquire a video of the user when it is detected that the user is speaking toward the electronic device. The apparatus further comprises a lip reading unit configured to run the lip reading model according to the parameters updated by the parameter updating unit to perform lip reading on the lip moving video segment to obtain a lip reading text.

30. The apparatus of claim 16, wherein: The video acquisition unit is further configured to acquire a video of the user when it is detected that the user is speaking toward the electronic device. The apparatus further comprises a lip reading unit configured to run the lip reading model according to the parameters updated by the parameter updating unit to perform lip reading on the lip moving video segment to obtain a lip reading text.

31. The apparatus of claim 17, wherein: The video acquisition unit is further configured to acquire a video of the user when it is detected that the user is speaking toward the electronic device. The apparatus further comprises a lip reading unit configured to run the lip reading module according to the parameters updated by the parameter updating unit to perform lip reading on the lip moving video to obtain a lip reading text.

32. The apparatus of claim 19, wherein: The video acquisition unit is further configured to acquire a video of the user when it detects that the user is speaking toward the electronic device. The apparatus further comprises a lip reading unit configured run the lip reading model according to the parameters updated by the parameter updating unit to perform lip reading to obtain a lip reading text.

33. The apparatus of claim 23, wherein: The video acquisition unit is further configured to acquire a video of the user when it detect that the user is speaking toward the electronic device. The apparatus further comprises a lip reading unit configured according to the parameters updated by the parameter updating unit to run the lip reading model to perform lip reading on the lip moving video segment to obtain a lip reading text.

34. The apparatus of claim 24, wherein the video acquisition unit is further configured to acquire a video of the user when it is detected that the user is speaking toward the electronic device; the lip reading apparatus further comprises a lip reading unit configured to run the lip reading model to perform lip reading on the lip movement video segment according to the parameters of the updated lip reading model to obtain lip reading text.

35. The apparatus of claim 26, wherein the video acquisition unit is further configured to acquire a video of the user when it is detected that the user is speaking toward the electronic device; the lip reading apparatus further comprises a lip reading unit configured to run the lip reading model to perform lip reading on the lip movement video segment according to the parameters of the updated lip reading model to obtain lip reading text.

36. The apparatus of claim 29, wherein the lip reading model comprises a general feature layer and a trainable layer, and the parameters of the lip reading model comprise trainable layer parameters and general feature layer parameters; the lip reading apparatus further comprises a preset information acquisition unit configured to acquire preset information of the user; the lip reading unit is specifically configured to acquire the trainable layer parameters associated with the preset information, load the trainable layer parameters and the preconfigured general feature layer parameters to run the lip reading model to perform lip reading on the lip movement video segment.

37. The apparatus of claim 36, wherein the preset information comprises a face ID; the preset information acquisition unit is specifically configured to perform face recognition on image frames in the video to obtain face feature data of the user, and query a face database to which a face ID corresponding to the face feature data is registered.

38. An electronic device, comprising: comprising: a processor; and a memory storing a computer program, the computer program causing the processor to execute the sample labeling method of any one of claims 1 to 5, the model training method of any one of claims 6 to 10, and / or the lip reading method of any one of claims 11 to 13 when executed by the processor.

39. A computer readable storage medium having stored thereon program instructions, wherein, the program instructions causing the computer to execute the sample labeling method of any one of claims 1 to 5, the model training method of any one of claims 6 to 10, and / or the lip reading method of any one of claims 11 to 13 when executed by the computer.

Images