A lip image processing method and device, electronic equipment, and storage medium

Abstract

This application discloses a lip image processing method, apparatus, electronic device, and storage medium. The method includes: acquiring a lip region image of a current video frame and its corresponding audio segment; extracting the phoneme sequence corresponding to the audio segment and generating a deterministic lip shape template based on the key point coordinates corresponding to the phoneme sequence in a rigid dictionary; wherein the rigid dictionary is used to characterize the correspondence between each phoneme and the geometric information of the standard lip shape; converting the deterministic lip shape template into a spatial gating mask; performing a strong suppression operation on features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and a current truncation threshold to obtain a suppressed visual feature map; and using the suppressed visual feature map to repair the lip region of the current video frame to obtain a repaired video frame corresponding to the current video frame.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a lip image processing method and apparatus, electronic device, and storage medium. Background Technology

[0002] When watching live online videos, hearing-impaired individuals rely heavily on lip-reading, meaning they depend heavily on the characters' lip movements. However, in low-bandwidth scenarios, video encoding can easily lead to blurred lips, increasing the difficulty of lip-reading. Therefore, it is necessary to repair the lips in each frame of the video.

[0003] Currently, generative lip restoration technology is mainly used, which employs lip generation models to generate restored lip images. However, in low-bandwidth or noisy environments, the model prioritizes visual naturalness, leading to "semantic illusions." For example, the model often tends to generate an average lip shape with a "slightly open mouth" to maintain naturalness, resulting in the loss of semantics such as "closed-mouth sounds."

[0004] Another approach uses a soft attention mechanism to fuse audio and video features for lip image restoration. However, the soft attention mechanism cannot block noise interference in non-lip regions, causing computational resources to be wasted on reconstructing irrelevant areas, ultimately resulting in unclear semantic information about the lip shape. Therefore, current methods aim for accurate real-time lip restoration in videos. Summary of the Invention

[0005] In view of the shortcomings of the prior art, this application provides a lip image processing method, apparatus, electronic device, and storage medium to solve the problem that the prior art cannot accurately repair lip images.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] The first aspect of this application provides a method for processing lip images, including:

[0008] Get the lip region image of the current video frame and its corresponding audio segment;

[0009] The phoneme sequence corresponding to the audio segment is extracted, and a deterministic lip shape template is generated based on the key point coordinates corresponding to the phoneme sequence in a rigid dictionary; wherein, the rigid dictionary is used to characterize the correspondence between the geometric information of each phoneme and the standard lip shape.

[0010] The deterministic lip template is converted into a spatial gating mask;

[0011] Based on the spatial gating mask and the current truncation threshold, a strong suppression operation is performed on the features outside the lip boundary in the visual feature map of the lip region image to obtain a suppressed visual feature map.

[0012] The lip region of the current video frame is repaired using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame.

[0013] Optionally, the above-described lip image processing method further includes:

[0014] Extract phoneme sequences and their corresponding standard lip shape images from a standard dataset;

[0015] Establish a normalized coordinate system for each of the aforementioned standard lip shape images;

[0016] Obtain the coordinates of each key point of the lip in each standard lip shape image under the normalized coordinate system to obtain the key point coordinates corresponding to each standard lip shape image;

[0017] The key point coordinates corresponding to the same standard lip shape image are averaged to obtain the key point coordinates of the standard lip shape for each phoneme.

[0018] A rigid dictionary is constructed using the key point coordinates of the standard lip shape corresponding to each phoneme.

[0019] Optionally, in the above-described lip image processing method, after obtaining the lip region image of the current video frame and its corresponding audio segment, the method further includes:

[0020] Establish a coordinate system for the lip region image that is identical to the keypoint coordinates in the rigid dictionary;

[0021] Map the key points of the lips in the lip region image to its coordinate system;

[0022] Extract the visual feature map of the lip region image.

[0023] Optionally, in the above-described lip image processing method, the step of performing a strong suppression operation on features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and the current truncation threshold to obtain a suppressed visual feature map includes:

[0024] Based on the current truncation threshold, the spatial gating mask is activated using an activation function to obtain the gating coefficient matrix;

[0025] Align the visual feature map of the lip region image with the gating coefficient matrix according to the coordinates of the key points of the lip region image;

[0026] The visual feature map of the aligned lip region image is spatially multiplied with the gating coefficient matrix to obtain the suppressed visual feature map.

[0027] Optionally, in the above-described lip image processing method, before performing a strong suppression operation on features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and the current truncation threshold to obtain the suppressed visual feature map, the method further includes:

[0028] The truncation threshold is adjusted according to the signal-to-noise ratio (SNR) of the current video frame to obtain the current truncation threshold; wherein, the larger the SNR, the smaller the truncation threshold.

[0029] Optionally, in the above-described lip image processing method, the step of extracting the phoneme sequence corresponding to the audio segment and generating a deterministic lip shape template based on the keypoint coordinates corresponding to the phoneme sequence in a rigid dictionary includes:

[0030] Extract the phoneme sequence corresponding to the audio segment;

[0031] Using a dynamic time warping algorithm, the key point coordinate sequence corresponding to the phoneme sequence is matched from the rigid dictionary;

[0032] A deterministic lip shape template is generated based on the key point coordinate sequence corresponding to the phoneme sequence.

[0033] Optionally, in the above-described lip image processing method, after repairing the lip region of the current video frame using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame, the method further includes:

[0034] Analyze the second-order difference of key points of the lips in the repaired video frame corresponding to consecutive multiple video frames;

[0035] If the second-order difference exceeds a set threshold, then each frame of the repaired video frame is smoothed and corrected.

[0036] A second aspect of this application provides a lip image processing apparatus, comprising:

[0037] The acquisition unit is used to acquire the lip region image of the current video frame and its corresponding audio segment;

[0038] The template generation unit is used to extract the phoneme sequence corresponding to the audio segment and generate a deterministic lip shape template based on the key point coordinates corresponding to the phoneme sequence in the rigid dictionary; wherein, the rigid dictionary is used to characterize the correspondence between the geometric information of each phoneme and the standard lip shape.

[0039] Template conversion unit, used to convert the deterministic lip template into a spatial gating mask;

[0040] The suppression unit is used to perform a strong suppression operation on the features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and the current truncation threshold, so as to obtain a suppressed visual feature map.

[0041] The repair unit is used to repair the lip region of the current video frame using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame.

[0042] Optionally, the lip image processing apparatus described above further includes:

[0043] The data extraction unit is used to extract phoneme sequences and their corresponding standard lip shape images from a standard dataset.

[0044] The first establishing unit is used to establish a normalized coordinate system for each of the standard lip shape images;

[0045] Obtain the coordinates of each key point of the lip in each standard lip shape image under the normalized coordinate system to obtain the key point coordinates corresponding to each standard lip shape image;

[0046] The key point coordinates corresponding to the same standard lip shape image are averaged to obtain the key point coordinates of the standard lip shape for each phoneme.

[0047] A rigid dictionary is constructed using the key point coordinates of the standard lip shape corresponding to each phoneme.

[0048] Optionally, the lip image processing apparatus described above further includes:

[0049] The second establishment unit is used to establish a coordinate system for the lip region image that is the same as the key point coordinates in the rigid dictionary;

[0050] A mapping unit is used to map key points of the lips in the lip region image to its coordinate system.

[0051] The feature extraction unit is used to extract the visual feature map of the lip region image.

[0052] Optionally, in the above-described lip image processing apparatus, the suppression unit includes:

[0053] An activation unit is used to activate the spatial gating mask based on the current truncation threshold using an activation function to obtain a gating coefficient matrix.

[0054] The alignment unit is used to align the visual feature map of the lip region image with the gating coefficient matrix according to the coordinates of the key points of the lip in the lip region image.

[0055] The suppression calculation unit is used to perform a spatial dot product between the visual feature map of the aligned lip region image and the gating coefficient matrix to obtain a suppressed visual feature map.

[0056] Optionally, the lip image processing apparatus described above further includes:

[0057] The threshold adjustment unit is used to adjust the truncation threshold according to the signal-to-noise ratio corresponding to the current video frame to obtain the current truncation threshold; wherein, the larger the signal-to-noise ratio, the smaller the truncation threshold.

[0058] Optionally, in the above-described lip image processing apparatus, the template generation unit includes:

[0059] A phoneme extraction unit is used to extract the phoneme sequence corresponding to the audio segment;

[0060] The matching unit is used to match the key point coordinate sequence corresponding to the phoneme sequence from the rigid dictionary using a dynamic time warping algorithm;

[0061] The generation unit is used to generate a deterministic lip shape template based on the key point coordinate sequence corresponding to the phoneme sequence.

[0062] Optionally, the lip image processing apparatus described above further includes:

[0063] The difference calculation unit is used to analyze the second-order difference of the key points of the lip in the repaired video frame corresponding to multiple consecutive video frames.

[0064] A smoothing processing unit is used to smooth and correct each frame of the repaired video frame when the second-order difference exceeds a set threshold.

[0065] A third aspect of this application provides an electronic device, comprising:

[0066] Memory and processor;

[0067] The memory is used to store programs;

[0068] The processor is used to execute the program, which, when executed, is specifically used to implement the lip image processing method as described in any of the above.

[0069] A fourth aspect of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the lip image processing method as described in any of the preceding claims.

[0070] This application provides a lip image processing method that acquires the lip region image of the current video frame and its corresponding audio segment, extracts the phoneme sequence corresponding to the audio segment, and generates a deterministic lip shape template based on the keypoint coordinates corresponding to the phoneme sequence in a rigid dictionary. The rigid dictionary is used to characterize the correspondence between each phoneme and the geometric information of the standard lip shape, thereby avoiding the influence of texture and introducing a deterministic geometric constraint mechanism. Then, the deterministic lip shape template is converted into a spatial gating mask, and based on the spatial gating mask and the current truncation threshold, a strong suppression operation is performed on the features outside the lip boundary in the visual feature map of the lip region image, resulting in a suppressed visual feature map. This utilizes the introduced geometric constraints to cut off the feature responses of the lip region and non-lip regions. Finally, the suppressed visual feature map is used to repair the lip region of the current video frame, resulting in a repaired video frame. This overcomes the bias of existing technologies, forcibly eliminates the randomness of the generation model according to the principle of semantic fidelity priority, cuts off the influence of non-lip regions through strong suppression, and ensures that the repaired lip shape strictly conforms to the physical definition of phonetics, thus achieving accurate repair of the lips of characters in the video. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0072] Figure 1 A flowchart illustrating a lip image processing method provided in this application embodiment;

[0073] Figure 2 This is a schematic diagram illustrating an example of extracting key point coordinates of the lips in an embodiment of this application;

[0074] Figure 3 A flowchart illustrating a method for constructing a rigid dictionary, as provided in an embodiment of this application;

[0075] Figure 4 A flowchart illustrating a preprocessing method for a lip region image provided in an embodiment of this application;

[0076] Figure 5 A flowchart illustrating a method for strongly suppressing the visual feature map of a lip region image, provided in an embodiment of this application;

[0077] Figure 6 A schematic diagram of the architecture of a lip image processing device provided in an embodiment of this application;

[0078] Figure 7 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0081] This application provides a method for processing lip images, such as... Figure 1 As shown, it includes the following steps:

[0082] S101. Obtain the lip region image of the current video frame and its corresponding audio segment.

[0083] Specifically, each frame in the video that needs repair is taken as the current video frame, and the lip region image in the current video frame is extracted in advance. At the same time, the audio segment corresponding to the current video frame is extracted from the audio data of the video.

[0084] Optionally, the audio within the range of N seconds before and N seconds after the current video frame's time point can be used as its corresponding audio segment.

[0085] S102. Extract the phoneme sequence corresponding to the audio segment, and generate a deterministic lip shape template based on the key point coordinates corresponding to the phoneme sequence in the rigid dictionary.

[0086] In this context, a rigid dictionary refers to a dictionary that stores immutable key-value pairs. In this embodiment, the rigid dictionary specifically comprises fields representing the correspondence between each phoneme and the geometric information of the standard lip shape. This rigid dictionary allows for strict constraint on the geometric information representing the lip shape corresponding to each phoneme. Optionally, the rigid dictionary may include pre-extracted key point coordinates of the standard lip shape corresponding to each phoneme.

[0087] It should be noted that, in this embodiment, in order to generate accurate lip shapes for the audio of the current video frame and achieve lip image restoration, a correspondence between phonemes and accurate lip shape information is generated in advance for each phoneme. Furthermore, to eliminate the randomness of texture generation at the data source, thereby ensuring the unique determinism of the constraint source and guaranteeing the accuracy of the generated lip image, the constructed rigid dictionary only stores the geometric information of the standard lip shape corresponding to each phoneme, eliminating the texture information of the lip image.

[0088] Optionally, in another implementation of this application, the rigid dictionary may specifically store only the coordinates of the geometric centroids of the standard lip shape corresponding to each phoneme, i.e., the coordinates of multiple key points, thus eliminating the texture information of the lip image while ensuring that the standard lip shape can be determined. For example, as... Figure 2 As shown in the right image, the data stores a pure geometric skeleton composed of key points, without any texture. Of course, it could also contain other geometric information such as lip contours.

[0089] Therefore, the sequence of phonemes corresponding to the audio segment is extracted, and the geometric information corresponding to the extracted phonemes is retrieved from the rigid field. The retrieved geometric information is then used to generate a deterministic lip shape target. Specifically, the key point coordinates corresponding to the extracted phonemes can be retrieved from the rigid field, and then the key point coordinates can be used to generate a deterministic lip shape template.

[0090] Optionally, in another embodiment of this application, a method for constructing a rigid dictionary is provided, such as... Figure 3 As shown, it includes:

[0091] S301. Extract phoneme sequences and their corresponding standard lip shape images from the standard dataset.

[0092] Specifically, images of the speaker accurately pronouncing each phoneme in the phoneme sequence can be collected in advance to obtain the standard lip shape image corresponding to each phoneme.

[0093] S302. Establish a normalized coordinate system for each standard lip shape image.

[0094] In order to eliminate scale differences caused by different face sizes and image focal lengths, it is necessary to construct a rigid field that includes normalized data. For this purpose, the same normalized coordinate system is established for each standard lip shape image.

[0095] The normalized coordinate system can be a coordinate system with a circle representing a position point that changes due to the lip shape variation, and a normalized unit distance representing a distance that changes due to the lip shape variation. Optionally, such as... Figure 2 As shown in the left figure, the normalized coordinate system can be a coordinate system with the tip of the nose as the origin and the distance between the eyes as the normalized unit, that is, the distance between the pupils of the eyes is defined as a normalized unit of 1.0.

[0096] S303. Obtain the coordinates of each key point of the lip in each standard lip shape image in the normalized coordinate system, and obtain the key point coordinates corresponding to each standard lip shape image.

[0097] S304. Average the coordinates of the key points corresponding to the same standard lip shape image to obtain the coordinates of the key points of the standard lip shape corresponding to each phoneme.

[0098] It should be noted that, to ensure the accuracy of the data in the rigid dictionary, multiple standard lip shape images are collected for each phoneme, and key points of the standard lip shape are extracted from each image. Then, the coordinates of the same key point are averaged to obtain the coordinates of the key points of the standard lip shape corresponding to that phoneme.

[0099] S305. Use the key point coordinates of the standard lip shape corresponding to each phoneme to form a rigid dictionary.

[0100] Optionally, in another embodiment of this application, one specific implementation of step S102 includes:

[0101] The phoneme sequence corresponding to the audio segment is extracted, and the key point coordinate sequence corresponding to the phoneme sequence is matched from the rigid dictionary using the dynamic time warping algorithm. A deterministic lip shape template is then generated based on the key point coordinate sequence corresponding to the phoneme sequence.

[0102] In order to improve the accuracy of the constructed lip shape template, the dynamic time warping algorithm is used to match the sequence of key point coordinates of each standard lip shape corresponding to the phoneme sequence, and a deterministic lip shape template is generated using the key point coordinate sequence corresponding to the phoneme sequence.

[0103] S103. Convert the deterministic lip template into a spatial gating mask.

[0104] To enable strong suppression of regions outside the lips using a deterministic lip template—that is, to identify the lip area within the lip image and apply strong suppression to that area—and thus achieve "strong isolation" under weak mesh conditions, forcibly severing the connection between lip semantic generation and the generation of other facial textures, avoiding the use of limited computing power for gradient blending outside the lip boundary, and focusing solely on lip optimization, the deterministic lip template is converted into a spatial gating mask.

[0105] Optionally, to achieve a smooth transition from the face to the lips and make the generated lips more accurate, the deterministic lip template can be converted into a spatial gate mask with steep edges. Here, steep edges refer to a transition zone decreasing from 1 to 0.

[0106] S104. Based on the spatial gating mask and the current truncation threshold, perform strong suppression on the features outside the lip boundary in the visual feature map of the lip region image to obtain the suppressed visual feature map.

[0107] Specifically, the lip region and non-lip region in the spatial gating mask are divided by a truncation threshold. Therefore, based on the spatial gating mask and the current truncation threshold, the lip features and non-lip features in the visual feature map of the lip region image can be determined, and strong suppression is applied to features outside the lip boundary, i.e., non-lip features, significantly reducing the response intensity of the non-lip region. Optionally, the features of the non-lip region can be forcibly set to a fixed value or attenuated below the truncation threshold. Therefore, the method provided in this embodiment breaks through the bias of existing technologies, proposes a semantic fidelity priority principle, and uses hard, strong suppression for processing, which differs from the soft weighting of conventional attention.

[0108] Optionally, to facilitate the suppression operation on the visual feature map of the lip region image, in another embodiment of this application, preprocessing of the lip region image can be performed before performing step S104. For example... Figure 4 As shown in the embodiment of this application, a preprocessing method for a lip region image includes:

[0109] S401. Establish a coordinate system for the lip region image that is identical to the coordinates of the key points in the rigid dictionary.

[0110] In order to align the visual feature map of the lip region image with the spatial gating mask constructed using the keypoint coordinates in the rigid dictionary—that is, to precisely "cover" the lips in the visual feature map to perform strong suppression—a coordinate system identical to the coordinate system of the keypoint coordinates in the rigid dictionary is established for the lip region image. Specifically, a normalized coordinate system can be constructed for the lip region image.

[0111] S402. Map the key points of the lips in the lip region image to its coordinate system.

[0112] Specifically, the key points of the lips in the lip region image are mapped to their coordinate system, which makes it easier to use the mapped key points as spatial anchors to center-align the mask with the visual feature map and perform scale matching.

[0113] S403. Extract the visual feature map of the lip region image.

[0114] Optionally, in another embodiment of this application, one specific implementation of step S104 includes:

[0115] The truncation threshold is adjusted based on the signal-to-noise ratio of the current video frame to obtain the current truncation threshold.

[0116] The higher the signal-to-noise ratio, the smaller the truncation threshold.

[0117] It should be noted that a dynamic truncation threshold is used to adaptively adjust the semantics and naturalness of the image based on network conditions. Specifically, the truncation threshold is adjusted according to the signal-to-noise ratio (SNR) of the current video frame. Optionally, the truncation threshold can be set according to the range of the SNR.

[0118] Optionally, when the signal-to-noise ratio (PSNR) is ≥22dB, the truncation threshold can be set to 0.3, thus retaining some texture naturalness while ensuring semantic accuracy when the PSNR is high. When the PSNR is <18dB, the truncation threshold can be set to 0.05, thus replacing the visual naturalness principle with the semantic fidelity priority principle to strengthen semantic fidelity.

[0119] Optionally, in another embodiment of this application, one specific implementation of step S104 is as follows: Figure 5 As shown, it includes:

[0120] S501. Based on the current truncation threshold, activate the spatial gating mask using an activation function to obtain the gating coefficient matrix.

[0121] Optionally, the HardSigmoid function or the ReLU function can be used to activate the difference between the mask and the threshold to generate a gating coefficient matrix.

[0122] S502. Align the visual feature map of the lip region image with the gating coefficient matrix according to the coordinates of the key points of the lip region image.

[0123] Specifically, based on the coordinates of the key points of the lips in the lip region image, the visual features belonging to the same key point are aligned with the gating coefficients, thereby aligning the visual feature map of the lip region image with the gating coefficient matrix.

[0124] S503. Perform a spatial dot product between the visual feature map of the aligned lip region image and the gating coefficient matrix to obtain the suppressed visual feature map.

[0125] Specifically, the visual feature map of the aligned lip region image is multiplied at the pixel level by the gating coefficient matrix to obtain the suppressed visual feature map.

[0126] Therefore, the formula for strong suppression of visual feature maps is:

[0127] F out =F in ⊙G(Mask,θ).

[0128] in, G is the visual feature map; G is the gating function used for activation; Mask is the spatial gating mask. This is the truncation threshold.

[0129] S105. Use the suppressed visual feature map to repair the lip region of the current video frame to obtain the repaired video frame corresponding to the current video frame.

[0130] Since the visual features of non-lip regions have been strongly suppressed in the suppressed visual feature map, the lip region of the person in the current video frame can be repaired based on the suppressed visual feature map, so as to obtain the repaired image, that is, the repaired video frame corresponding to the current video frame.

[0131] Optionally, the lip region of the current video frame can be repaired directly based on the suppressed visual feature map using a generative network, resulting in a repaired video frame. Alternatively, the lip image of the current video frame can be repaired based on the suppressed visual feature map using a generative network, and then the repaired lip image can be fused with the current video frame to obtain the repaired video frame.

[0132] Optionally, in order to ensure that the generated results meet physiological limits and obtain results that are more in line with reality, in another embodiment of this application, after performing step S105, the following can be further performed:

[0133] The second-order difference of key points of the lips in the repaired video frames corresponding to consecutive video frames is analyzed, and when the second-order difference exceeds a set threshold, each repaired video frame is smoothed and corrected.

[0134] Optionally, if the second-order difference exceeds a threshold range of 0.12 to 0.18 normalization units, smoothing correction can be performed. Alternatively, a one-dimensional Kalman filter or moving average algorithm can be used to perform temporal low-pass filtering on the lip feature vectors of multiple consecutive repaired video frames to forcibly reduce abrupt high-frequency jitter and make it conform to the smooth trajectory of muscle movement.

[0135] This application provides a lip image processing method. It acquires the lip region image of the current video frame and its corresponding audio segment, extracts the phoneme sequence corresponding to the audio segment, and generates a deterministic lip shape template based on the keypoint coordinates corresponding to the phoneme sequence in a rigid dictionary. The rigid dictionary characterizes the correspondence between each phoneme and the geometric information of the standard lip shape, thereby avoiding the influence of texture and introducing a deterministic geometric constraint mechanism. The deterministic lip shape template is then converted into a spatially gated mask with steep edge attributes. Based on the spatially gated mask and the current truncation threshold, a strong suppression operation is performed on features outside the lip boundary in the visual feature map of the lip region image, resulting in a suppressed visual feature map. This utilizes the introduced geometric constraints to cut off the feature responses of the lip region and non-lip regions. Finally, the lip region of the current video frame is repaired using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame. This breaks the bias of the existing technology. According to the principle of semantic fidelity priority, the randomness of the generation model is forcibly eliminated by strong suppression, and the influence of non-lip regions is cut off. This ensures that the repaired lip shape strictly conforms to the physical definition of phonetics, thus achieving accurate repair of the lips of the characters in the video.

[0136] More specifically, compared with the prior art, the lip image processing method provided in this application has the following specific beneficial effects:

[0137] 1. Breaking away from the biases of existing technologies, this application completely solves the problem of "semantic illusion" under low bandwidth: Existing generative models and soft attention mechanisms typically prioritize visual naturalness in low bandwidth or high noise environments, leading to the loss of key semantics such as "closed-mouth sounds" (i.e., semantic illusion). This application breaks away from this technological bias by proposing a "semantic fidelity priority" principle. By constructing a "rigid dictionary" containing only geometric centroid coordinates (keypoint coordinates), the randomness of texture generation is eliminated, and a spatial gating mask with steep edge attributes is introduced to replace the conventional "soft weighting" with "hard strong suppression," forcibly cutting off noise interference in non-lip regions. Testing has verified that under extreme conditions such as weak networks, such as when the signal-to-noise ratio (PSNR) of the current video is <18dB, the method provided in this application can maintain a forced lip-reading accuracy of over 78% and a closed-mouth sound error rate of less than 5%, effectively improving the accuracy of lip restoration and significantly enhancing the lip-reading experience for the hearing impaired.

[0138] 2. Extremely lightweight design ensures extremely low latency on mobile devices in weak network environments: The "rigid dictionary" in this application only stores the key point coordinates of the standard lip shape of each phoneme (pure geometric skeleton), completely eliminating high-dimensional lip image texture information, resulting in a very small dictionary size and resource consumption. Furthermore, by combining efficient gating coefficient matrix space for dot product operations, computational power consumption is significantly reduced, allowing the entire lip repair inference latency to be stably controlled within 150ms, perfectly meeting the real-time requirements of mobile live streaming and other scenarios.

[0139] 3. Adapting to the network environment to achieve the best balance between semantics and visual naturalness: This application innovatively proposes a mechanism for dynamically adjusting the truncation threshold based on the signal-to-noise ratio. Specifically, when network conditions are good (high signal-to-noise ratio), the threshold is lowered to retain some texture naturalness; when the network is congested (low signal-to-noise ratio), semantic fidelity is prioritized, and a stricter strong suppression threshold is adopted, thus realizing intelligent dynamic adaptation of the processing strategy.

[0140] 4. Conforms to the physical laws of muscle movement, eliminating high-frequency jitter in the repaired image: This application analyzes the second-order difference of key lip points in multiple consecutive repaired video frames and performs smoothing correction with the support of one-dimensional Kalman filtering or moving average algorithms. This mechanism forcibly reduces the abrupt high-frequency jitter caused by single-frame repair, ensuring that the generated lip shape is strictly limited by the physiological limits and smooth trajectory of human facial muscle movement, further improving the smoothness and visual appeal of the repaired video.

[0141] Another embodiment of this application provides a lip image processing apparatus, such as... Figure 6 As shown, it includes:

[0142] The acquisition unit 601 is used to acquire the lip region image of the current video frame and its corresponding audio segment.

[0143] The template generation unit 602 is used to extract the phoneme sequence corresponding to the audio segment and generate a deterministic lip shape template based on the key point coordinates corresponding to the phoneme sequence in a rigid dictionary. The rigid dictionary is used to characterize the correspondence between the geometric information of each phoneme and the standard lip shape.

[0144] Template conversion unit 603 is used to convert a deterministic lip template into a spatial gating mask.

[0145] Suppression unit 604 is used to perform strong suppression operation on features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and the current truncation threshold, so as to obtain a suppressed visual feature map.

[0146] Repair unit 605 is used to repair the lip region of the current video frame using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame.

[0147] Optionally, in another embodiment of the lip image processing apparatus provided in this application, the apparatus further includes:

[0148] The data extraction unit is used to extract phoneme sequences and their corresponding standard lip-shape images from a standard dataset.

[0149] The first unit is used to establish a normalized coordinate system for each standard lip shape image.

[0150] Obtain the coordinates of each key point of the lip in each standard lip shape image in the normalized coordinate system to obtain the key point coordinates corresponding to each standard lip shape image.

[0151] The key point coordinates corresponding to the same standard lip shape image are averaged to obtain the key point coordinates of the standard lip shape for each phoneme.

[0152] A rigid dictionary is constructed using the key point coordinates of the standard lip shape corresponding to each phoneme.

[0153] Optionally, in another embodiment of the lip image processing apparatus provided in this application, the apparatus further includes:

[0154] The second establishment unit is used to establish a coordinate system for the lip region image that is the same as the coordinates of the key points in the rigid dictionary.

[0155] The mapping unit is used to map the key points of the lips in the lip region image to its coordinate system.

[0156] The feature extraction unit is used to extract visual feature maps of the lip region image.

[0157] Optionally, in another embodiment of the lip image processing apparatus provided in this application, the suppression unit includes:

[0158] The activation unit is used to activate the spatial gating mask based on the current truncation threshold using an activation function to obtain the gating coefficient matrix.

[0159] Alignment units are used to align the visual feature map of the lip region image with the gating coefficient matrix according to the coordinates of the key points of the lip in the lip region image.

[0160] The suppression calculation unit is used to perform a spatial dot product between the visual feature map of the aligned lip region image and the gating coefficient matrix to obtain the suppressed visual feature map.

[0161] Optionally, in another embodiment of the lip image processing apparatus provided in this application, the apparatus further includes:

[0162] The threshold adjustment unit is used to adjust the truncation threshold according to the signal-to-noise ratio (SNR) of the current video frame to obtain the current truncation threshold. The higher the SNR, the lower the truncation threshold.

[0163] Optionally, in another embodiment of the lip image processing apparatus provided in this application, the template generation unit includes:

[0164] The phoneme extraction unit is used to extract the phoneme sequence corresponding to an audio segment.

[0165] The matching unit is used to match the key point coordinate sequence corresponding to the phoneme sequence from the rigid dictionary using the dynamic time warping algorithm.

[0166] The generation unit is used to generate a deterministic lip shape template based on the key point coordinate sequence corresponding to the phoneme sequence.

[0167] Optionally, in another embodiment of the lip image processing apparatus provided in this application, the apparatus further includes:

[0168] The difference calculation unit is used to analyze the second-order difference of key points of the lips in the repaired video frame corresponding to multiple consecutive video frames.

[0169] The smoothing unit is used to smooth and correct each frame of the repaired video when the second-order difference exceeds a set threshold.

[0170] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.

[0171] Another embodiment of this application provides an electronic device, such as... Figure 7 As shown, it includes:

[0172] Memory 701 and processor 702.

[0173] The memory 701 is used to store the program.

[0174] The processor 702 is used to execute the program stored in the memory 701. When the program is executed, it is specifically used to implement the lip image processing method provided in any of the above embodiments.

[0175] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the lip image processing method provided in any of the above embodiments.

[0176] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0177] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing lip images, characterized in that, include: Get the lip region image of the current video frame and its corresponding audio segment; The phoneme sequence corresponding to the audio segment is extracted, and a deterministic lip shape template is generated based on the key point coordinates corresponding to the phoneme sequence in a rigid dictionary; wherein, the rigid dictionary is used to characterize the correspondence between the geometric information of each phoneme and the standard lip shape. The deterministic lip template is converted into a spatial gating mask; Based on the spatial gating mask and the current truncation threshold, a strong suppression operation is performed on the features outside the lip boundary in the visual feature map of the lip region image to obtain a suppressed visual feature map. The lip region of the current video frame is repaired using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame.

2. The method according to claim 1, characterized in that, Also includes: Extract phoneme sequences and their corresponding standard lip shape images from a standard dataset; Establish a normalized coordinate system for each of the aforementioned standard lip shape images; Obtain the coordinates of each key point of the lip in each standard lip shape image under the normalized coordinate system to obtain the key point coordinates corresponding to each standard lip shape image; The key point coordinates corresponding to the same standard lip shape image are averaged to obtain the key point coordinates of the standard lip shape for each phoneme. A rigid dictionary is constructed using the key point coordinates of the standard lip shape corresponding to each phoneme.

3. The method according to claim 1, characterized in that, After obtaining the lip region image of the current video frame and its corresponding audio segment, the method further includes: Establish a coordinate system for the lip region image that is identical to the keypoint coordinates in the rigid dictionary; Map the key points of the lips in the lip region image to its coordinate system; Extract the visual feature map of the lip region image.

4. The method according to claim 3, characterized in that, The step of performing a strong suppression operation on features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and the current truncation threshold to obtain a suppressed visual feature map includes: Based on the current truncation threshold, the spatial gating mask is activated using an activation function to obtain the gating coefficient matrix; Align the visual feature map of the lip region image with the gating coefficient matrix according to the coordinates of the key points of the lip region image; The visual feature map of the aligned lip region image is spatially multiplied with the gating coefficient matrix to obtain the suppressed visual feature map.

5. The method according to claim 1, characterized in that, Before performing a strong suppression operation on features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and the current truncation threshold to obtain the suppressed visual feature map, the process further includes: The truncation threshold is adjusted according to the signal-to-noise ratio (SNR) of the current video frame to obtain the current truncation threshold; wherein, the larger the SNR, the smaller the truncation threshold.

6. The method according to claim 1, characterized in that, The step of extracting the phoneme sequence corresponding to the audio segment and generating a deterministic lip shape template based on the keypoint coordinates corresponding to the phoneme sequence in a rigid dictionary includes: Extract the phoneme sequence corresponding to the audio segment; Using a dynamic time warping algorithm, the key point coordinate sequence corresponding to the phoneme sequence is matched from the rigid dictionary; A deterministic lip shape template is generated based on the key point coordinate sequence corresponding to the phoneme sequence.

7. The method according to claim 1, characterized in that, After repairing the lip region of the current video frame using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame, the method further includes: Analyze the second-order difference of key points of the lips in the repaired video frame corresponding to consecutive multiple video frames; If the second-order difference exceeds a set threshold, then each frame of the repaired video frame is smoothed and corrected.

8. A lip image processing device, characterized in that, include: The acquisition unit is used to acquire the lip region image of the current video frame and its corresponding audio segment; The template generation unit is used to extract the phoneme sequence corresponding to the audio segment and generate a deterministic lip shape template based on the key point coordinates corresponding to the phoneme sequence in the rigid dictionary; wherein, the rigid dictionary is used to characterize the correspondence between the geometric information of each phoneme and the standard lip shape. Template conversion unit, used to convert the deterministic lip template into a spatial gating mask; The suppression unit is used to perform a strong suppression operation on the features outside the lip boundary in the visual feature map of the lip region image based on the spatial gating mask and the current truncation threshold, so as to obtain a suppressed visual feature map. The repair unit is used to repair the lip region of the current video frame using the suppressed visual feature map to obtain the repaired video frame corresponding to the current video frame.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement the lip image processing method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, is used to implement the lip image processing method as described in any one of claims 1 to 7.

Images