Method, device and equipment for determining voice keyword in video and storage medium

Abstract

The present disclosure relates to a method, device, equipment and storage medium for determining a speech keyword in a video. The method for determining a speech keyword in a video comprises: obtaining a video, the video comprising human voice and a face image of a user corresponding to the human voice; identifying lip key points of the face image included in each video frame in the video; performing speech recognition on the human voice, and in response to identifying a target speech keyword, determining a start timestamp and an end timestamp corresponding to the target speech keyword; obtaining a target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp and the end timestamp; and matching the target mouth opening amplitude sequence with a preset mouth opening amplitude sequence corresponding to the target speech keyword, and if the target mouth opening amplitude sequence matches the preset mouth opening amplitude sequence, determining that the human voice emitted by the user comprises the target speech keyword. Through the present disclosure, the accuracy and reliability of target speech keyword identification and determination are improved.

Description

Technical Field

[0001] This disclosure relates to the field of video analytics, and in particular to methods, apparatus, devices, and storage media for determining speech keywords in videos. Background Technology

[0002] With the development of artificial intelligence, computer vision, and speech recognition technologies, video analytics is widely used in various fields, such as finance and security. In the financial industry, especially in business transactions or remote video witnessing scenarios, users are often required to express specific keywords, such as "I know" or "I agree," as a basis for identity verification or business authorization. However, in this process, related technologies largely rely on speech recognition technology, using speech-to-text algorithms to identify the content of keywords and their corresponding timestamps to confirm the user's response. But in practical applications, complex scenarios such as noise interference can arise, affecting the accuracy and reliability of identity verification and business authorization.

[0003] To address the aforementioned issues, relevant technologies have incorporated facial landmark detection, which accurately identifies key facial features to effectively detect the lip area. This improvement enhances the accuracy of identity verification and business authorization. However, current facial landmark detection technologies are typically designed for actions with fixed opening and closing amplitudes, such as the eyes, and their descriptions of lip movements are overly simplistic. This leads to significant shortcomings in the calculation and analysis of lip movement amplitude, making it difficult to accurately reflect the diverse opening and closing states and amplitudes of the lips, and hindering precise matching of keyword actions. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, device, and storage medium for determining speech keywords in a video.

[0005] According to a first aspect of the present disclosure, a method for determining speech keywords in a video is provided, comprising: acquiring a video, the video including a human voice and a facial image of a user corresponding to the human voice; identifying lip key points of the facial images included in each video frame of the video; performing speech recognition on the human voice in the video, and determining a start timestamp and an end timestamp corresponding to the target speech keyword in response to the identification of the target speech keyword; obtaining a target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp, and the end timestamp, wherein the mouth opening amplitude sequence corresponds to different speech keywords; matching the target mouth opening amplitude sequence with a preset mouth opening amplitude sequence corresponding to the target speech keyword, and if the target mouth opening amplitude sequence matches the preset mouth opening amplitude sequence, determining that the human voice emitted by the user includes the target speech keyword.

[0006] In one embodiment, obtaining the target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp, and the end timestamp includes: determining the mouth opening amplitude based on the lip key points to obtain the user's mouth opening amplitude sequence, wherein the mouth opening amplitude is used to reflect the opening and closing state of the user's lips; and extracting the target mouth opening amplitude sequence that generates the target speech keyword from the mouth opening amplitude sequence based on the start timestamp and the end timestamp.

[0007] In one embodiment, obtaining the target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp, and the end timestamp includes: extracting a video segment from the video that generates the target speech keyword based on the start timestamp and the end timestamp; identifying the lip key points of each video frame within the video segment; and determining the target mouth opening amplitude sequence of the user in the video based on the lip key points.

[0008] In one embodiment, the mouth opening amplitude is determined as follows: all external lip key points and all internal lip key points included in the lip key points are determined; a first polygon formed by all external lip key points is determined, and a first area of ​​the first polygon is determined; a second polygon formed by all internal lip key points is determined, and a second area of ​​the second polygon is determined; the ratio of the second area to the first area is determined as the mouth opening amplitude.

[0009] In one embodiment, determining the mouth opening amplitude based on the lip key points includes: determining all auxiliary key points and acquiring a portion of inner lip key points, the portion of inner lip key points including a left inner lip key point, a middle inner lip key point, and a right inner lip key point; obtaining multiple sets of distances based on all auxiliary key points, the left inner lip key point, the middle inner lip key point, and the right inner lip key point, the multiple sets of distances including a first left distance and a second left distance between each auxiliary key point and the left inner lip key point, a first middle distance and a second middle distance between each auxiliary key point and the middle inner lip key point, and a first right distance and a second right distance between each auxiliary key point and the right inner lip key point; if any of the multiple sets of distances does not meet the distance condition, the mouth opening amplitude is set to 0.

[0010] In one embodiment, matching the target mouth opening amplitude sequence with a preset mouth opening amplitude sequence corresponding to the target speech keyword includes: calculating the dynamic time warping distance between each target mouth opening amplitude sequence and the preset mouth opening amplitude sequence corresponding to the target speech keyword, and determining the minimum dynamic time warping distance from the calculated dynamic time warping distances to obtain a set of minimum dynamic time warping distances; determining the minimum dynamic time warping distance in the set of minimum dynamic time warping distances; if the minimum dynamic time warping distance is less than or equal to a distance threshold, then determining that the target speech keyword is included in the user's voice in the video; if the minimum dynamic time warping distance is greater than the distance threshold, then determining that the target speech keyword is not included in the user's voice in the video.

[0011] In one embodiment, determining the start and end timestamps corresponding to the target speech keyword includes: recognizing the video and obtaining a time information sequence, the time information sequence including the start and end timestamps of the text content within the video; if the text content includes the target speech keyword, then determining the initial start and initial end timestamps of the target speech keyword; determining the N frames before the initial start timestamp as the start timestamp, and determining the M frames after the initial end timestamp as the end timestamp, wherein N and M are positive integers.

[0012] In one implementation, the video is acquired in a financial business scenario.

[0013] In one embodiment, after determining that the target speech keyword is included in the voice emitted by the user in the video, the method further includes: outputting at least one of the following: target speech keyword; start timestamp and end timestamp; target mouth opening amplitude sequence; business action, the business action including whether to trigger business logic.

[0014] In one embodiment, the method further includes: if the target mouth opening amplitude sequence is determined to be invalid within the target time period of the start timestamp and the end timestamp, then the target mouth opening amplitude sequence is not matched with the preset mouth opening amplitude sequence corresponding to the target speech keyword; the target mouth opening amplitude sequence being invalid includes at least one of the following situations: the mouth opening amplitude is continuously 0; the mouth opening amplitude is continuously less than the amplitude threshold; any one of the multiple distances does not meet the distance condition.

[0015] In one embodiment, the preset mouth opening amplitude sequence is pre-trained as follows: video clips of different users expressing the target speech keyword in different scenarios are collected; the mouth opening amplitude sequence in the video clips is extracted to obtain the mouth opening amplitude sequence corresponding to the different users in different scenarios; the length of the mouth opening amplitude sequence corresponding to the different users in different scenarios is unified to obtain the preset mouth opening amplitude sequence corresponding to the target speech keyword.

[0016] According to a second aspect of the present disclosure, an apparatus for determining speech keywords in a video is provided, comprising: an acquisition unit for acquiring a video, the video including a human voice and a facial image of a user corresponding to the human voice; a recognition unit for recognizing lip key points of the facial images included in each video frame of the video; a determination unit for performing speech recognition on the human voice in the video, and determining a start timestamp and an end timestamp corresponding to the target speech keyword in response to recognizing the target speech keyword; a processing unit for obtaining a target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp, and the end timestamp, wherein the mouth opening amplitude sequence corresponds to different speech keywords; and a matching unit for matching the target mouth opening amplitude sequence with a preset mouth opening amplitude sequence corresponding to the target speech keyword, wherein if the target mouth opening amplitude sequence matches the preset mouth opening amplitude sequence, it is determined that the human voice emitted by the user includes the target speech keyword.

[0017] In one embodiment, based on the lip key points, the start timestamp, and the end timestamp, the processing unit obtains the user's target mouth opening amplitude sequence in the video in the following manner: determining the mouth opening amplitude based on the lip key points to obtain the user's mouth opening amplitude sequence, wherein the mouth opening amplitude is used to reflect the user's lip opening and closing state; and extracting the target mouth opening amplitude sequence that generates the target speech keyword from the mouth opening amplitude sequence based on the start timestamp and the end timestamp.

[0018] In one embodiment, based on the lip key points, the start timestamp, and the end timestamp, the processing unit obtains the user's target mouth opening amplitude sequence in the video in the following manner: based on the start timestamp and the end timestamp, a video segment generating the target speech keyword is extracted from the video; the lip key points of each video frame within the video segment are identified; and based on the lip key points, the user's target mouth opening amplitude sequence in the video is determined.

[0019] In one embodiment, the mouth opening amplitude is determined as follows: all external lip key points and all internal lip key points included in the lip key points are determined; a first polygon formed by all external lip key points is determined, and a first area of ​​the first polygon is determined; a second polygon formed by all internal lip key points is determined, and a second area of ​​the second polygon is determined; the ratio of the second area to the first area is determined as the mouth opening amplitude.

[0020] In one embodiment, the determining unit determines the mouth opening amplitude based on the lip key points in the following manner: determining all auxiliary key points and acquiring a portion of inner lip key points, the portion of inner lip key points including the left inner lip key point, the middle inner lip key point, and the right inner lip key point; based on all auxiliary key points, the left inner lip key point, the middle inner lip key point, and the right inner lip key point, obtaining multiple sets of distances, the multiple sets of distances including a first left distance and a second left distance between each auxiliary key point and the left inner lip key point, a first middle distance and a second middle distance between each auxiliary key point and the middle inner lip key point, and a first right distance and a second right distance between each auxiliary key point and the right inner lip key point; if any of the multiple sets of distances does not meet the distance condition, the mouth opening amplitude is set to 0.

[0021] In one embodiment, the matching unit matches the target mouth opening amplitude sequence with the preset mouth opening amplitude sequence corresponding to the target speech keyword in the following manner: It calculates the dynamic time warping distance between each target mouth opening amplitude sequence and the preset mouth opening amplitude sequence corresponding to the target speech keyword, and determines the minimum dynamic time warping distance from the calculated dynamic time warping distances to obtain a set of minimum dynamic time warping distances; it determines the minimum dynamic time warping distance in the set of minimum dynamic time warping distances; if the minimum dynamic time warping distance is less than or equal to a distance threshold, it determines that the user's voice in the video includes the target speech keyword; if the minimum dynamic time warping distance is greater than the distance threshold, it determines that the user's voice in the video does not include the target speech keyword.

[0022] In one embodiment, the determining unit determines the start and end timestamps corresponding to the target speech keyword in the following manner: the video is identified to obtain a time information sequence, the time information sequence including the start and end timestamps of the text content within the video; if the text content includes the target speech keyword, then the initial start and initial end timestamps of the target speech keyword are determined; the N frames before the initial start timestamp are determined as the start timestamp, and the M frames after the initial end timestamp are determined as the end timestamp, where N and M are positive integers.

[0023] In one implementation, the video is acquired in a financial business scenario.

[0024] In one embodiment, after determining that the target speech keyword is included in the user's voice in the video, the determining unit is further configured to: output at least one of the following: target speech keyword; start timestamp and end timestamp; target mouth opening amplitude sequence; business action, the business action including whether to trigger business logic.

[0025] In one embodiment, the matching unit is further configured to: if the target mouth opening amplitude sequence is determined to be invalid within the target time period of the start timestamp and the end timestamp, then not match the target mouth opening amplitude sequence with the preset mouth opening amplitude sequence corresponding to the target speech keyword; the target mouth opening amplitude sequence being invalid includes at least one of the following situations: the mouth opening amplitude is continuously 0; the mouth opening amplitude is continuously less than the amplitude threshold; any one of the multiple distances does not meet the distance condition.

[0026] In one embodiment, the preset mouth opening amplitude sequence is pre-trained as follows: video clips of different users expressing the target speech keyword in different scenarios are collected; the mouth opening amplitude sequence in the video clips is extracted to obtain the mouth opening amplitude sequence corresponding to the different users in different scenarios; the length of the mouth opening amplitude sequence corresponding to the different users in different scenarios is unified to obtain the preset mouth opening amplitude sequence corresponding to the target speech keyword.

[0027] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the method for determining voice keywords in a video as described in the first aspect or any embodiment of the first aspect.

[0028] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the method for determining voice keywords in a video as described in the first aspect or any embodiment of the first aspect.

[0029] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: This disclosure obtains the target mouth opening amplitude sequence of the user in the video by using the lip key points of the facial images included in each video frame, as well as the start timestamp and end timestamp corresponding to the target speech keywords. On the one hand, introducing lip key points to determine the target mouth opening amplitude sequence improves the stability and continuity of the target mouth opening amplitude sequence estimation; on the other hand, combining the start timestamp and end timestamp corresponding to the target speech keywords ensures complete capture of the start and end of the mouth shape changes during the user's pronunciation process, reducing the probability of truncating mouth shape fragments.

[0030] This disclosure matches a target mouth opening amplitude sequence with a preset mouth opening amplitude sequence corresponding to a target speech keyword to determine whether the user's voice contains the target speech keyword. It is understood that the preset mouth opening amplitude sequence includes various mouth opening amplitudes; therefore, matching the determined target mouth opening amplitude sequence with the preset mouth opening amplitude sequence can improve the matching success rate. Thus, determining speech keywords in video using the above method can improve the accuracy and reliability of target speech keyword recognition and determination, meeting user needs and enhancing the user experience.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0033] Figure 1 This is a flowchart illustrating a method for determining speech keywords in a video according to an exemplary embodiment.

[0034] Figure 2 This is a schematic diagram illustrating key points of a facial structure according to an exemplary embodiment.

[0035] Figure 3 This is a flowchart illustrating a method for determining a target mouth opening amplitude sequence according to an exemplary embodiment.

[0036] Figure 4 This is a flowchart illustrating a method for determining a target mouth opening amplitude sequence according to an exemplary embodiment.

[0037] Figure 5 This is a flowchart illustrating a method for determining the mouth opening amplitude according to an exemplary embodiment.

[0038] Figure 6 This is a flowchart illustrating a method for determining the mouth opening amplitude according to an exemplary embodiment.

[0039] Figure 7 This is a schematic diagram illustrating a mouth opening amplitude sequence according to an exemplary embodiment.

[0040] Figure 8 This is a schematic diagram illustrating a mouth opening amplitude sequence according to an exemplary embodiment.

[0041] Figure 9 This is a flowchart illustrating a method for determining a timestamp according to an exemplary embodiment.

[0042] Figure 10 This is a flowchart illustrating a matching method according to an exemplary embodiment.

[0043] Figure 11 This is a schematic diagram illustrating a DTW time alignment according to an exemplary embodiment.

[0044] Figure 12 This is a flowchart illustrating a training method for a preset mouth opening amplitude sequence according to an exemplary embodiment.

[0045] Figure 13 This is a schematic diagram illustrating a storage structure according to an exemplary embodiment.

[0046] Figure 14 This is a block diagram illustrating a device for determining speech keywords in a video according to an exemplary embodiment.

[0047] Figure 15 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure. Detailed Implementation

[0048] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0049] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0050] Video analytics is widely used in various fields, such as analyzing whether users verbally express specific keywords. These keywords could be phrases like "I know" or "I agree." Related technologies often rely on speech recognition or facial landmark detection.

[0051] In speech recognition, speech-to-text technology is typically used to determine the timestamps of spoken keywords. That is, the identification of spoken keywords is usually based on the recognition results of speech-to-text, without considering the matching relationship between the lip opening and closing actions in the video and the timing of the keywords. In complex scenarios such as the presence of multiple people or noise interference, this can easily lead to misjudgment or inaccurate recognition.

[0052] In facial landmark detection, the Dlib open-source library is used. Dlib is a machine learning-based open-source library for facial landmark detection that can accurately identify 68 key points on the face, especially in the lip region. This provides a data foundation for further analysis of lip movements. However, the calculation and analysis of lip movement amplitude is still relatively crude, often relying on the Eye Aspect Ratio (EAR) function from eye opening and closing detection methods. However, the EAR function typically suffers from the following problems: it can only roughly distinguish between open and closed states, cannot effectively represent the complexity and individual differences of lip movements, and is difficult to achieve accurate action matching for specific speech keywords; the EAR function is essentially designed for eye movements with fixed opening and closing amplitudes, and its description of lip movement changes is too simplistic, failing to accurately reflect the diverse opening and closing states and amplitudes of the lips, and cannot effectively describe the diversity and irregularity of lip opening and closing movements, resulting in large quantification errors in lip movement amplitude and poor detection accuracy; there are obvious individual differences in lip opening and closing movements, and traditional general-purpose EAR functions do not consider these differences and lack a normalization design to adapt to individual changes, resulting in poor feature extraction stability and an inability to accurately identify lip movement features corresponding to specific keywords.

[0053] Furthermore, the aforementioned reliance on speech recognition technology to confirm keywords, and on speech-to-text confirmation to determine whether the keywords have appeared, without combining visual lip movements for secondary confirmation, makes it susceptible to environmental noise, overlapping speech from multiple people, and other factors, leading to a decrease in the accuracy of speech keyword confirmation.

[0054] In view of this, this disclosure proposes a method for determining speech keywords in videos, which fully considers the variability of lip movements and individual differences, refines the characteristics of lip movements, and combines speech recognition technology to achieve accurate matching of speech keywords.

[0055] Figure 1 This is a flowchart illustrating a method for determining speech keywords in a video according to an exemplary embodiment. Figure 1 As shown, the method includes steps S11 to S15.

[0056] In step S11, a video is acquired, which includes human voices and the facial image of the user who made the voices.

[0057] In this embodiment of the disclosure, the video can be acquired in real time in a financial business scenario, or it can be acquired in batches while offline.

[0058] In step S12, the key points of the lips in the face images included in each video frame are identified.

[0059] It should be noted that face detection in each video frame can be performed using lightweight and highly accurate face detection networks, such as the Dual-Branch Face Detection Network (DBFace) or the single-stage dense face localization algorithm (RetinaFace). The face detection network disclosed herein is merely illustrative and not intended to be limiting.

[0060] In some embodiments, a pre-trained 68-key facial landmark model from the Dlib open-source library can be used to accurately obtain the coordinates of 68 key points in the facial image of each video frame. These 68 key points can cover areas such as the chin contour, eyebrows, eyes, nose, and mouth.

[0061] For example, in combination Figure 2 Explaining 68 key points. For example... Figure 2 As shown, Figure 2 This includes 68 key points of the facial structure. It can be understood that the lip key points include 20 points numbered 49-68. It should be noted that the two-dimensional coordinates of these key points are stored in a structured manner for use in calculating mouth opening amplitude and time series analysis.

[0062] In step S13, speech recognition is performed on the human voice in the video. In response to the recognition of target speech keywords, the start timestamp and end timestamp corresponding to the target speech keywords are determined.

[0063] In this embodiment of the disclosure, the target voice keyword can be a voice keyword such as "I know" or "I agree". The human voice in the video may or may not include the target voice keyword. If the target voice keyword is identified in the video, the start and end timestamps of the target voice keyword can be determined based on speech-to-text technology.

[0064] In step S14, the target mouth opening amplitude sequence of the user in the video is obtained based on the lip key points, the start timestamp, and the end timestamp.

[0065] In this embodiment of the disclosure, the lip key points can reflect the user's mouth opening amplitude, and the target mouth opening amplitude sequence related to the target speech keyword can be obtained based on the start timestamp and end timestamp of the target speech keyword.

[0066] It should be noted that because the mouth opening amplitude corresponds to different characters in each phonetic keyword, the mouth opening amplitude sequence corresponds to different phonetic keywords. Furthermore, the mouth opening amplitude sequence is a sequence that represents the relationship between mouth opening amplitude and time.

[0067] In step S15, the target mouth opening amplitude sequence is matched with the preset mouth opening amplitude sequence corresponding to the target speech keyword. If the target mouth opening amplitude sequence matches the preset mouth opening amplitude sequence, it is determined that the user's voice includes the target speech keyword.

[0068] Understandably, the preset mouth opening amplitude sequence includes mouth opening amplitude sequences for different speech keywords, different users, different accents, and different scenarios. Based on the matching of the target mouth opening amplitude sequence and the preset mouth opening amplitude sequence, it is possible to accurately determine whether the user's voice contains the target speech keywords, thereby reducing the possibility of misjudgment.

[0069] In this embodiment of the disclosure, since the target mouth opening amplitude sequence is determined based on the lip key points and the start and end timestamps corresponding to the target speech keywords, that is, considering both the user's mouth opening amplitude and the time when the target speech keywords are spoken, a sequence representing the change of mouth opening amplitude over time is obtained, which improves the accuracy and robustness of target speech keyword recognition from a temporal perspective.

[0070] In this embodiment of the disclosure, the target mouth opening amplitude sequence can be determined using the following two methods, based on lip key points, start timestamps, and end timestamps. The following is combined with... Figure 3 One method for determining the target mouth opening amplitude sequence is described.

[0071] Figure 3 This is a flowchart illustrating a method for determining a target mouth opening amplitude sequence according to an exemplary embodiment. Figure 3 As shown, the method includes steps S21 to S22.

[0072] In step S21, the mouth opening amplitude is determined based on the key points of the lips to obtain the user's mouth opening amplitude sequence.

[0073] In this embodiment of the disclosure, the mouth opening amplitude is used to reflect the opening and closing state of the user's lips. Lip key points are acquired from each video frame within the video, and the mouth opening amplitude of each video frame is determined, thereby obtaining the mouth opening amplitude sequence corresponding to the video.

[0074] It is understood that the video may or may not contain the target speech keyword. The mouth opening sequence corresponding to the video may include the mouth opening amplitude corresponding to the user speaking the target speech keyword, or it may include the mouth opening amplitude corresponding to the user speaking other words.

[0075] In step S22, based on the start timestamp and the end timestamp, the target mouth opening amplitude sequence that generates the target speech keyword is extracted from the mouth opening amplitude sequence.

[0076] In this embodiment of the disclosure, since the start timestamp and end timestamp are the start and end times of the corresponding target speech keyword in the video, the corresponding time period is determined and extracted from the mouth opening amplitude sequence based on this time period, so as to obtain the target mouth opening amplitude sequence corresponding to the target speech keyword.

[0077] In this embodiment of the disclosure, in a sequence of mouth opening amplitudes that includes all mouth opening amplitudes, a target mouth opening amplitude sequence is determined based on the start timestamp and the end timestamp, which can accurately filter out the mouth opening amplitudes corresponding to the target speech keywords.

[0078] The following combination Figure 4 Another method for determining the target mouth opening amplitude sequence is described.

[0079] Figure 4 This is a flowchart illustrating a method for determining a target mouth opening amplitude sequence according to an exemplary embodiment. Figure 4 As shown, the method includes steps S31 to S33.

[0080] In step S31, based on the start timestamp and end timestamp, a video segment that generates the target speech keywords is extracted from the video.

[0081] In this embodiment of the disclosure, the audio portion of the video is obtained, and the start and end timestamps of the target speech keywords are determined based on the audio. A video segment corresponding to the start and end timestamps is extracted from the video, and this video segment is used as the video segment for generating the target speech keywords.

[0082] In step S32, the lip key points of each video frame within the video segment are identified.

[0083] In step S33, the target mouth opening amplitude sequence of the user in the video is determined based on lip key points.

[0084] In this embodiment of the disclosure, the mouth opening amplitude of each video frame is determined based on the lip key points of each video frame in the video segment, thereby obtaining the mouth opening amplitude sequence of the video segment, that is, the target mouth opening amplitude sequence.

[0085] In this embodiment, a target mouth opening amplitude sequence can be obtained by determining the mouth opening amplitude sequence, start timestamp, and end timestamp, and then extracting the target mouth opening amplitude sequence based on the start and end timestamps. This method can be used in non-real-time processing scenarios, such as historical video review or offline batch processing. Alternatively, a video segment generating the target speech keywords can be extracted from the video based on the start and end timestamps, thereby obtaining the target mouth opening amplitude sequence based on the user's mouth opening amplitude in the video segment. This method can be used for video determination, such as in scenarios requiring rapid response to target speech key points.

[0086] In this embodiment of the disclosure, regardless of which of the two methods described above is used, it is necessary to determine the mouth opening amplitude based on lip key points, thereby obtaining a target mouth opening amplitude sequence related to the target speech keywords. For example... Figure 2 As shown, in common methods for measuring lip opening and closing, many studies have attempted to draw on the EAR structure from blink detection, using the length ratio between key points as the basis for estimating mouth opening amplitude. For example, the ratio of the Euclidean distance between key points on the upper and lower edges of the inner lip (such as No. 63 and No. 67) to the distance between key points on both sides of the corners of the mouth (such as No. 61 and No. 65) is selected to reflect the mouth opening amplitude.

[0087] However, in practical applications, the lip structure differs from that of the eyes. Due to individual speaking habits, the lips may be in a "pursed" or "closed" state. In this case, although the distance between the two ends of the lips (such as points 67 and 63) is not zero, the direction may be reversed, resulting in the actual closed mouth state appearing as an "open mouth" numerically. Furthermore, the amplitude of mouth opening varies significantly among individuals during speech expression, making the length ratio-based calculation method sensitive to noise and individual differences, resulting in poor stability. For example, the ratio of the distance between key points 63 and 67 to the distance between key points 61 and 65 is used as the EAR value. However, in the case of "pursed lips," the distance from key point 67 is smaller than the distance from key point 63 to key point 34. In other words, the direction vectors of the two key points are reversed, but the distance is a scalar and remains positive. Therefore, the EAR value can appear as an "open mouth," but the actual mouth is closed ("pursed lips"), leading to an error in judging whether the mouth opening amplitude is actually open or not.

[0088] Therefore, this disclosure uses the ratio of areas, rather than the ratio of lengths, to determine the user's mouth opening range. The following is combined with... Figure 2 and Figure 5 The method for determining the mouth opening range is explained. Figure 5 This is a flowchart illustrating a method for determining mouth opening amplitude according to an exemplary embodiment. Figure 5 As shown, the method includes steps S41 to S44.

[0089] In step S41, all external lip key points and all internal lip key points included in the lip key points are determined.

[0090] In this embodiment, the outer lip region includes 12 key points (49-60) used to delineate the overall contour of the lips. The inner lip region includes 8 key points (61-68) used to delineate the inner boundary between the upper and lower lips. Since points 50, 52, 54, 56, 58, and 60 lie on the lines connecting points 49 and 51, 51 and 53, 53 and 55, 55 and 57, 57 and 59, and 59 and 49 respectively, points 49, 51, 53, 55, 57, and 59 can be selected as all the key points for the outer lip. Points 61-68 are selected as all the key points for the inner lip.

[0091] In step S42, the first polygon formed by all the key points of the outer lip is determined, and the first area of ​​the first polygon is determined.

[0092] In step S43, the second polygon formed by all the key points of the inner lip is determined, and the second area of ​​the second polygon is determined.

[0093] In this embodiment of the disclosure, the polygon formed by numbers 49, 51, 53, 55, 57, and 59 is designated as the first polygon, and the polygon formed by numbers 61 to 68 is designated as the second polygon. The first area of ​​the first polygon and the second area of ​​the second polygon are then determined.

[0094] In step S44, the ratio of the second area to the first area is determined as the mouth opening amplitude.

[0095] In this embodiment of the disclosure, the mouth opening amplitude can be determined using the Mouth Opening Area Ratio (MOAR) function. The formula for calculating the mouth opening amplitude function is as follows:

[0096]

[0097] MOAR refers to the degree of mouth opening. A inner The second area, A, is formed by all the key points of the inner lip (numbers 61-68).outer The first area, which is composed of all the key points of the outer lip (49, 51, 53, 55, 57, 59), is used as a normalization benchmark. Both can be calculated using the Gaussian polygon area calculation formula (shoelace formula).

[0098] It should be noted that since the outer lip contour always exists, the denominator A outer The value is always greater than 0, so there is no issue of dividing by zero. Furthermore, since the second area of ​​the inner lip is smaller than the first area, the mouth opening amplitude is limited by the proportional relationship, and the range of the mouth opening amplitude is MOAR∈[0,1), exhibiting good controllability. Moreover, compared to the length ratio calculation method, since the area formed is less sensitive to keypoint jitter than the line segment formed by two points, this area ratio function (MOAR) causes less disturbance to the keypoint position than the aspect ratio, exhibiting strong robustness, high stability, and distinguishability.

[0099] In some embodiments, the mouth opening amplitude is determined based on lip key points, resulting in a mouth opening amplitude sequence. This sequence can be understood as filtering video frames of the user's mouth opening and assembling the mouth opening amplitudes from all such frames. Therefore, accurate judgment of the user's open and closed mouth states is necessary. Considering special cases such as lip key point jitter and lip pursing, the judgment of the closed mouth state is added. In other words, the accuracy of identifying the closed mouth state is improved, and misjudgments caused by key point detection errors are eliminated.

[0100] For example, to achieve the above effect, auxiliary key points can be introduced for judgment, such as... Figure 6 As shown, Figure 6 This is a flowchart illustrating a method for determining mouth opening amplitude according to an exemplary embodiment, the method including steps S51 to S53.

[0101] In step S51, all auxiliary key points are determined, and some inner lip key points are obtained.

[0102] Among them, some key points of the inner lip include key points of the left inner lip (62 and 68), key points of the middle inner lip (63 and 67), and key points of the right inner lip (64 and 67). Auxiliary key points include key points 32, 34, and 36.

[0103] In step S52, multiple sets of distances are obtained based on all auxiliary keypoints, the left inner lip keypoint, the middle inner lip keypoint, and the right inner lip keypoint.

[0104] Among them, multiple sets of distances include the first left distance and the second left distance between each auxiliary key point and the left inner lip key point, the first middle distance and the second middle distance between each auxiliary key point and the middle inner lip key point, and the first right distance and the second right distance between each auxiliary key point and the right inner lip key point.

[0105] For example, the first left distance is the distance between numbers 62 and 32, and the second left distance is the distance between numbers 68 and 32. The first middle distance is the distance between numbers 63 and 34, and the second middle distance is the distance between numbers 67 and 34. The first right distance is the distance between numbers 64 and 36, and the second right distance is the distance between numbers 66 and 36.

[0106] In step S53, if any of the multiple distance sets does not meet the distance condition, the mouth opening amplitude is set to 0.

[0107] It's important to note that while a user's head can face in various directions, the different parts of the face are relatively fixed. Therefore, the distance from the lower part of the inner lip (key points 66, 67, and 68) to the nose (key points 32, 34, and 36) is greater than the distance from the upper part of the inner lip (key points 62, 63, and 64) to the nose (key points 32, 34, and 36). Thus, the opposite will only occur if the "closed mouth" state (e.g., pursed lips) and key point detection are inaccurate.

[0108] In other words, the distance condition can be understood as follows: the second left distance is greater than the first left distance, the second middle distance is greater than the second middle distance, and the second right distance is greater than the first middle distance. If any of these three sets of distances is less than a certain value, it indicates an abnormal keypoint orientation, which is identified as an anomaly. This could be due to a pursed lip or a keypoint recognition error. In this case, the MOAR value is set to allow for subsequent filtering based on this value.

[0109] It is understandable that the distance condition not being met can be at least one of the following: the second left distance is less than the first left distance, the second middle distance is less than the first middle distance, or the second right distance is less than the first right distance.

[0110] In this embodiment, since the facial structure maintains a relatively stable geometric relationship at different angles, the lower edge points (66, 67, 68) of the inner lip should normally be far from the midline of the bridge of the nose (32, 34, 36) and its upper edge points (62, 63, 64); otherwise, it is an abnormal state. By introducing the above judgment, the misjudgment problem of the traditional length ratio method is overcome, the accuracy of mouth opening amplitude estimation is improved, and the ability to avoid misjudgment of abnormal states is enhanced.

[0111] In this embodiment, multiple lip key points are used to form a stable polygon, and the mouth opening amplitude is determined based on the area of ​​the polygon, which improves the stability, continuity and anti-interference ability of mouth opening amplitude estimation, and provides a reliable feature basis for subsequent speech-visual matching.

[0112] In some embodiments, the acquired video's original mouth opening amplitude sequence is as follows: Figure 7 As shown, the horizontal axis represents the number of frames, and the vertical axis represents the MOAR value. The original mouth opening amplitude sequence refers to the mouth opening amplitude sequence encompassing all video frames from the beginning to the end of the video. The mouth opening amplitude sequence is the sequence that determines the user's mouth opening state. The target mouth opening amplitude sequence is the mouth opening amplitude sequence that matches the target speech keyword. The original mouth opening amplitude sequence includes the mouth opening amplitude sequence, and the mouth opening amplitude sequence includes the target mouth opening amplitude sequence.

[0113] Figure 7 The mid-peak can be understood as the local maximum value of the mouth opening amplitude (MOAR value), corresponding to the high amplitude movement of the mouth opening and closing during pronunciation. For example, the peak may be associated with the pronunciation of a specific syllable (such as the vowel / open phoneme). For example, when saying "I know," the pronunciation of "dao" ( / ao / ) will trigger a noticeable MOAR peak.

[0114] In some embodiments, Figure 7 The calculated original mouth opening amplitude sequence still exhibits significant jitter and fluctuations between frames. This jitter primarily stems from slight head movements in the video, which affect the stability of facial landmark detection, causing minor shifts in lip landmarks between consecutive frames, thus leading to fluctuations in the mouth opening amplitude MOAR value. Therefore, to improve the stability and practical usability of the MOAR sequence, filtering can be introduced into the original mouth opening amplitude sequence, i.e., smoothing the MOAR value calculated for each frame. For example, the following two filtering methods can be used in combination:

[0115] (1) Threshold Filtering:

[0116] If the MOAR value of the current frame is less than a preset threshold (e.g., 0.05), it is set to zero and regarded as a "closed mouth" state, thereby filtering out invalid opening fluctuations caused by micro-vibrations.

[0117] (2) Mean Filtering:

[0118] A sliding window approach is used, employing a double-ended queue to store the historical MOAR values ​​of the most recent 5 frames, and then performing weighted smoothing on the current frame's value. Specifically, the current frame's MOAR value is replaced by the average of its value with the previous 4 frames, reducing the interference of local jumps on the overall trend judgment.

[0119] After the above two-stage filtering process, the resulting mouth opening amplitude MOAR time series exhibits smoother and more consistent trends, as shown in... Figure 8 As shown. Figure 8 The MOAR time series curve after the application of filtering is shown, which effectively suppresses the pseudo-fluctuations caused by lip key point jitter, further enhancing the discriminability of mouth opening features in the time dimension, and providing high-quality data support for subsequent target speech keyword mouth shape matching.

[0120] In this embodiment of the disclosure, in order to extract the target mouth opening amplitude sequence corresponding to the target speech keyword from the video, speech-to-text technology can be used to analyze the audio part of the video to accurately obtain the time interval of the appearance of the target speech keyword, that is, to determine the start timestamp and end timestamp.

[0121] Figure 9 This is a flowchart illustrating a method for determining a timestamp according to an exemplary embodiment. For example... Figure 9 As shown, the method includes steps S61 to S63.

[0122] In step S61, the video is identified to obtain a time information sequence, which includes the start and end timestamps of the text content in the video.

[0123] In this embodiment, a speech recognition model can be used to process the audio in the video, transcribing the speech content into a time information sequence in real time. The time information sequence includes the video text content, such as each character or word, and the corresponding timestamp of each character or word in the audio. The speech recognition model can be a Whisper model with high accuracy and strong noise resistance, supporting the return of word-level timestamp information. Alternatively, it can be a speech recognition engine (Wav2Vec2.0+CTC structure) capable of offline deployment and accurate time annotation.

[0124] In step S62, if the text content includes target speech keywords, the initial start timestamp and initial end timestamp of the target speech keywords are determined.

[0125] In this embodiment of the disclosure, if a target voice keyword (such as "I know", "I agree") set according to business requirements is detected, then the time interval [t] of the target voice keyword is obtained in the time information sequence. start ,t end ]. Among them, t start t is the initial start timestamp. end This is the initial and end timestamp.

[0126] In step S63, the N frames before the initial start timestamp are determined as the start timestamp, and the M frames after the initial end timestamp are determined as the end timestamp.

[0127] In [t] start ,t end Based on this, to fully cover the start and end phases of the lip movement, the N frames before the initial start timestamp can be defined as the start timestamp, and the M frames after the initial end timestamp can be defined as the end timestamp. Here, N and M are positive integers. For example, the time interval can be extended forward by 1 second and backward by 1 second to form a complete analysis window. That is:

[0128] [t start -1,t end +1]. Where, t start -1 represents the start timestamp, t end +1 is the end timestamp.

[0129] In this embodiment of the disclosure, it can be based on [t] start -1,t end +1], extract the corresponding start from the mouth opening amplitude sequence.

[0130] The target mouth opening amplitude sequence with timestamp and end timestamp:

[0131] MOAR sequence =[MOAR[t start -1],MOAR[t end +1]]

[0132] Among them, MOAR sequence For the target mouth opening amplitude sequence, MOAR[t start [-1] represents the mouth opening amplitude value corresponding to the start timestamp, MOAR[t] end +1] is the mouth opening amplitude value corresponding to the end timestamp.

[0133] The target mouth opening amplitude sequence fully covers the dynamic changes of the lips when the user utters the target speech keyword, which facilitates direct input for subsequent mouth shape recognition and matching based on time sequence patterns.

[0134] In this embodiment of the disclosure, the target mouth opening amplitude sequence reflects the dynamics of lip opening and closing during the user's expression of target speech keywords. To determine whether the target mouth opening amplitude sequence matches the typical mouth shape pattern of the target speech keywords, the target mouth opening amplitude sequence and a preset mouth opening amplitude sequence can be time-aligned and similarity calculated using a dynamic time warping algorithm (DTW). Figure 10 This is a flowchart illustrating a matching method according to an exemplary embodiment. For example... Figure 10 As shown, the method includes steps S71 to S73b.

[0135] In step S71, the DTW distance is calculated one by one between the target mouth opening amplitude sequence and the preset mouth opening amplitude sequence corresponding to the target speech keyword, and the minimum DTW distance is determined from the calculated DTW distances to obtain the minimum DTW distance set.

[0136] In this embodiment, video samples of multiple individuals naturally expressing target speech keywords are collected. The mouth opening amplitude sequence corresponding to the target speech keyword is extracted and normalized (including frame length alignment and amplitude normalization). The normalized mouth opening amplitude sequence is then used as a preset mouth opening amplitude sequence. The preset mouth opening amplitude sequence can be represented as: T = {T1, T2, ..., T...} k}. Where each T i =[t1,t2,...,t n ] represents the mouth opening amplitude sequence of a sample, where each t j (j∈[1,n]) is the normalized mouth opening amplitude value (MOAR value) of the j-th frame in the mouth opening amplitude sequence.

[0137] It should be noted that the DTW algorithm uses dynamic programming to calculate the optimal matching path and minimize the cumulative distance, thereby solving the temporal offset problem, such as the difference in speaking speed.

[0138] In this embodiment, the target mouth opening amplitude sequence is discretized, that is, the mouth opening amplitude value corresponding to each frame is obtained, resulting in the discretized target mouth opening amplitude sequence Q = [q1, q2, ..., q m The DTW distance is calculated for Q and T respectively. The recursive formula for the DTW distance calculation is as follows:

[0139]

[0140] Where D(i,j) represents the minimum cumulative distance, i.e., the minimum DTW distance, when the i-th frame of the target mouth opening amplitude sequence Q is aligned with the j-th frame of the preset mouth opening amplitude sequence T. The initial boundary condition is D(0,0) = 0, and the remaining undefined parts are initialized to positive infinity. This recursive process can be viewed as finding the shortest path from the top left to the bottom right on a two-dimensional time axis, such that the input sequence can be stretched or compressed on the time axis for optimal alignment with the template, as shown below. Figure 11 As shown. q1, q2, q3…q i …q m The mouth opening amplitudes in the target mouth opening amplitude sequence are t1, t2, t3…t. j …t n This refers to the mouth opening amplitude in the preset mouth opening amplitude sequence.

[0141] In this embodiment of the disclosure, the DTW distance is calculated one by one based on the target mouth opening amplitude sequence and the preset mouth opening amplitude sequence corresponding to the target speech keyword, resulting in multiple minimum DTW distances. These multiple minimum DTW distances are then combined into a minimum DTW distance set, denoted as Δ={D1,D2,...,D...} k},D i =DTW(Q,T) i ).

[0142] In step S72, the minimum DTW distance in the minimum DTW distance set is determined.

[0143] In this embodiment of the disclosure, the minimum DTW distance D is determined from the minimum DTW distance set Δ. min D min The input sequence is compared with a preset threshold θ. Based on the comparison result, it is determined whether to execute step S73a or step S73b. Here, θ is the similarity threshold for judging whether the input sequence matches the standard template sequence. It can be set empirically based on the statistical distribution of DTW matching distance within the preset mouth opening amplitude sequence, or obtained through cross-validation.

[0144] In step S73a, if the minimum DTW distance is less than or equal to the distance threshold, it is determined that the target speech keywords are included in the human voice emitted by the user in the video.

[0145] In this embodiment of the disclosure, if there exists at least one minimum DTW distance satisfying D min If the lip movements in the current video frame are consistent with the target speech keyword, the MOAR(Q) of the current frame is determined to be a successful match with the preset mouth opening amplitude sequence of the target speech keyword, and it can be considered that the user has genuinely expressed the target speech keyword in the video.

[0146] In step S73b, if the minimum DTW distance is greater than the distance threshold, it is determined that the target speech keyword is not included in the human voice emitted by the user in the video.

[0147] In this embodiment of the disclosure, if D min If the value is >θ, it is considered that the user has not expressed the target speech keywords or that the expression is unclear.

[0148] In this embodiment, lip-shape patterns are aligned and robustly recognized along the temporal dimension, effectively resisting temporal disturbances caused by variations in speaking speed, rhythm, and individual differences in mouth opening amplitude, and providing a decision-making basis for subsequent action confirmation and result output. Furthermore, considering that different individuals may have certain amplitude and rhythm differences when expressing the same target speech keyword, a preset mouth opening amplitude sequence is used for matching to improve the system's adaptability to individual pronunciation diversity.

[0149] In this embodiment of the disclosure, the preset mouth opening amplitude sequence is obtained through pre-collection and training. Figure 12 This is a flowchart illustrating a training method for a preset mouth opening amplitude sequence according to an exemplary embodiment. Figure 12 As shown, the method includes steps S81 to S83.

[0150] In step S81, video clips of different users expressing target speech keywords in different scenarios are collected.

[0151] The video clips include the extent to which the user opens their mouth when uttering the target spoken keywords.

[0152] In step S82, the mouth opening amplitude sequence is extracted from the video clip to obtain the mouth opening amplitude sequence corresponding to different users in different scenarios.

[0153] In step S83, the length of the mouth opening amplitude sequence corresponding to different users in different scenarios is unified to obtain the preset mouth opening amplitude sequence corresponding to the target speech keyword.

[0154] In this embodiment of the disclosure, video clips of different users expressing target speech keywords in different scenarios are collected and processed to obtain a preset mouth opening amplitude sequence, which can improve the matching tolerance and further improve the matching success rate of the system using this method in diverse populations.

[0155] In this embodiment of the disclosure, after determining that the user's voice in the video includes the target speech keyword, the system can output a structured matching result, including but not limited to the following: the target speech keyword (e.g., "I know"); the start and end timestamps, corresponding to the time window in which the user expresses the target speech keyword in the video; the matching confidence score, which can be obtained by linear normalization based on the minimum DTW distance; the target mouth opening amplitude sequence, which may include descriptive indicators such as the maximum value and average value of the target mouth opening amplitude sequence; and business actions, including whether business logic is triggered, such as compliance approval or authorization confirmation.

[0156] In this embodiment of the disclosure, the matching results can provide complete institutionalized data support for the integration of systems such as finance, security, or interaction.

[0157] In this embodiment of the disclosure, to further enhance the robustness of the method, an automatic abnormal state discrimination mechanism can be introduced. That is, if the target mouth opening amplitude sequence is determined to be invalid within the target time period of the start and end timestamps, the target mouth opening amplitude sequence is not matched with the preset mouth opening amplitude sequence corresponding to the target speech keyword, and a "mismatch" or "rejection" status is output.

[0158] Among them, the invalidity of the target mouth opening amplitude sequence includes at least one of the following situations: the mouth opening amplitude is continuously 0, such as when there is occlusion, mouth closure, or prolonged inactivity; the mouth opening amplitude is continuously less than the amplitude threshold, which can be considered as an insignificant mouth opening amplitude, such as when the curve of the mouth opening amplitude sequence fluctuates very little or appears as a horizontal line; any one of the multiple distances fails to meet the distance condition, that is, the distance direction between the inner lip key point and the bridge of the nose is incorrect, triggering the closed mouth state to 0.

[0159] In this embodiment of the disclosure, by designing an automatic abnormal state discrimination mechanism, invalid input situations such as blurred mouth shape or continuous closed mouth are rejected, which can ensure that the output results of the method are safe and reliable.

[0160] In this embodiment, the method for determining voice keywords in a video and the matching results can be integrated into various real-world business scenarios through system interfaces. Examples include remote authorization systems for banks, video customer compliance verification systems, and multimodal verification modules in intelligent voice interaction.

[0161] In this embodiment of the disclosure, the matching result determined by the mouth opening amplitude is used for reliable judgment and standardized output, which provides a direct basis for subsequent business processes or user feedback, thereby improving the practical value and implementation of the voice mouth shape matching system.

[0162] The following embodiments of this disclosure will use "I know" as an example to illustrate the method for determining voice keywords in a video.

[0163] S1: Face detection and key point extraction.

[0164] In this embodiment, the goal is to accurately extract the temporal features of the user's lip opening and closing movements during the natural expression of the word "I know," in order to support subsequent visual keyword matching and judgment. To this end, face detection and facial landmark extraction are first performed on each video frame in the user's video. The specific operation process is as follows:

[0165] S1.1 Video Frame-by-Frame Processing and Face Detection:

[0166] The test video was decoded frame by frame at a rate of 25 frames per second, and each frame was input into the face detection model for detection. In this embodiment, the lightweight and high-precision face detector DBFace was used to locate all faces in the image and output the corresponding bounding boxes. This model can run in real time on edge devices and has good multi-face detection capabilities.

[0167] S1.2 extracts 68 facial feature points based on the Dlib open-source library:

[0168] For the detected face regions, the 68-point facial landmark detection model provided by Dlib is used to model the facial structure of the face within each bounding box, outputting a sequence of two-dimensional coordinates for the landmarks. Each set of coordinates corresponds to a specific facial region, as shown below:

[0169] Chin and facial contour (numbers 1-17); eyebrows, bridge of nose, and eye area (numbers 18-48); lip area (numbers 49-68); outer lip (numbers 49-60), 12 points in total, used to describe the outer edge of the lip shape; inner lip (numbers 61-68), 8 points in total, used to depict the upper and lower inner boundaries of the open mouth area. The keypoint layout is as follows: Figure 2 As shown, it conforms to the official Dlib standard point definition.

[0170] S1.3 Structured storage of key information:

[0171] The two-dimensional coordinates of the detected lip keypoints (numbers 49-68) in each frame are recorded in a structured manner and associated with the corresponding frame number, forming a structure like this: Figure 13 The structure is as follows: the sequence of lip key points extracted from consecutive video frames will serve as the basic input for subsequent calculation of mouth opening amplitude.

[0172] The key objective of this step is to provide accurate, stable, and continuous structural visual data for modeling lip movements during the expression of "I know," ensuring high reliability of MOAR calculations in subsequent steps. For facial expressions, Dlib keypoints maintain high stability even with frontal facial views and slight facial rotations, making them suitable for processing video data in non-cooperative states.

[0173] S2: Mouth opening amplitude quantification and smoothing.

[0174] After completing the stable extraction of facial key points in S1, this embodiment further designs a mouth opening amplitude quantization function MOAR based on area ratio for the dynamics of lip opening and closing during the expression of the word "I know", and combines a multi-level filtering mechanism to perform sequence smoothing processing on the amplitude.

[0175] S2.1 Design the quantization function MOAR for mouth opening amplitude:

[0176] In common practices, the degree of mouth opening is often estimated by referencing the EAR (Eye Aspect Ratio) structure used in blink detection, such as by estimating the degree of mouth opening using the ratio of the vertical distance between keypoints 63 and 67 to the horizontal distance between keypoints 61 and 65. However, this embodiment finds that this method has the following problems:

[0177] (1) The structure of the lips is more complex than that of the eyes. The direction vector may be reversed when the lips are pursed, causing misjudgment as "open mouth"; (2) The range of mouth opening varies greatly from person to person. The length ratio method is sensitive to disturbances and has poor stability.

[0178] Therefore, this disclosure proposes a MOAR (MouthOpeningAreaRatio) calculation method based on area ratio.

[0179] Numerator: Area of ​​the closed polygon formed by the key points of the inner lip (numbered 61-68); Denominator: Area of ​​the hexagon formed by the key points of the outer lip (49, 51, 53, 55, 57, 59); Formula definition:

[0180] Among them, A inner For the second area, A outer Let MOAR be the area of ​​the mouth opening. The area of ​​the polygon is calculated using the Gaussian polygon area formula (shoelace formula). The shoelace formula is as follows: Where (x) n+1 ,y n+1 ) = (x1, y1).

[0181] In this sequence, i = 1 to n are the key points “61 to 68” and “49, 51, 53, 55, 57, 59”, where x is the x-coordinate and y is the y-coordinate. After arranging the key points in clockwise or counterclockwise order, the first and second areas are calculated accordingly.

[0182] This disclosure also incorporates a structure-assisted judgment mechanism to improve the accuracy of "closed mouth" state recognition. In some embodiments, the following three sets of key point distances are compared: the distance from points 62 and 68 to point 32 on the bridge of the nose; the distance from points 63 and 67 to point 34; and the distance from points 64 and 66 to point 36.

[0183] If the distance from the lower lip key point to the bridge of the nose point in any group is less than the corresponding point on the lip, it indicates an abnormality in key point detection or a "pursed lips" situation. In this case, the MOAR value is set to 0, which is considered as a closed mouth.

[0184] The MOAR function has the following advantages: the denominator is always greater than 0, avoiding division by zero errors; the output value range is limited to [0,1), making the value controllable; it is less sensitive to disturbances at key points, resulting in strong stability. Figure 7 Within frames 240 to 280, the initial target mouth opening amplitude sequence during the user's process of saying "I know" is represented, which is also the time series curve.

[0185] S2.2 Smoothing MOAR using a filtering function:

[0186] from Figure 7It is known that although MOAR can reflect the temporal changes of mouth opening, there are still significant jitters between frames, affecting matching performance. The main reasons are facial micro-movements and key point detection errors. Therefore, a two-stage filtering mechanism can be adopted: threshold filtering and mean filtering. Threshold filtering: If the MOAR value of a frame is lower than the threshold (e.g., 0.05), it is directly set to 0, and non-significant mouth opening movements are excluded. Mean filtering: A sliding window of length 5 is constructed to store the MOAR values ​​of the latest 5 frames, and the MOAR output of the current frame is the arithmetic mean of all values ​​within the window. This smoothing strategy can not only filter out abnormal spikes, but also suppress local fluctuations, making the lip opening and closing trajectory smooth and representative.

[0187] In this embodiment of the disclosure, such as Figure 8 As shown, frames 240 to 280 display the filtered sequence of the target mouth opening amplitude during the same user saying "I know," which is the MOAR time series curve. The curve shows smooth changes and clear opening and closing rhythm, greatly improving the matching compatibility with the template sequence.

[0188] S3: Keyword timestamp location and sequence extraction.

[0189] In this embodiment of the disclosure, after obtaining a stable target mouth opening amplitude sequence, in order to accurately extract the corresponding mouth shape change segments during the user's expression "I know," a speech recognition model is introduced to locate keywords and delineate the sequence time period. The delineation can be performed in the following manner:

[0190] S3.1 Speech Recognition and Timestamp Acquisition:

[0191] In this embodiment, the audio segments extracted from the video are input into the speech-to-text module and processed using the Whisper model. This model not only provides high-accuracy Chinese recognition but also returns precise start and end timestamp information for each word.

[0192] Location of target speech keywords in the mapped image frame S3.2:

[0193] In this embodiment of the disclosure, Figure 8 In the smoothed MOAR curve shown, frames 240 to 280 correspond to the complete process of the user uttering "I know": around frame 240, the MOAR value begins to rise rapidly, representing the opening of the lip movement when pronouncing "I"; after approaching frame 250, MOAR decreases, reflecting the brief closing of the mouth when pronouncing "know"; around frame 260, MOAR rises again to a peak, corresponding to the pronunciation of "dao"; subsequently, MOAR drops rapidly, indicating the end of the pronunciation and the closure of the mouth.

[0194] Therefore, in this embodiment, the start frame (Frame_start)=215 (1 second in advance, i.e., 25 frames subtracted) and the end frame (Frame_end)=305 (1 second delayed) can be used as the frame-level time segment of "I know" in this embodiment.

[0195] S3.3 Construction of MOAR subsequences corresponding to keywords:

[0196] In the embodiment of the present disclosure, the above frame interval is used as the target time period composed of the start timestamp and the end timestamp. The target mouth opening amplitude range is expressed as: Q = [MOAR 215 , MOAR 216 , ..., MOAR 315

[0197] During a typical utterance process, this sequence visually presents the following characteristics:

[0198] The character "Wo (I)" (the syllable contains / w / + vowel): MOAR rises;

[0199] The character "Zhi (know)" (the initial consonant / zh / ): MOAR drops back briefly;

[0200] The character "Dao (know)" (the vowel / ao / ): MOAR rises to the maximum value again;

[0201] Closing the mouth at the end: MOAR drops rapidly to zero.

[0202] The target mouth opening amplitude sequence completely describes the lip opening and closing dynamics of the target voice keyword "Wo Zhi Dao (I know)", and provides high-quality input for subsequent DTW matching.

[0203] S4: Dynamic time warping and mouth shape matching based on DTW.

[0204] After obtaining the mouth opening amplitude time sequence of the target voice keyword "Wo Zhi Dao (I know)", the embodiment of the present disclosure uses the dynamic time warping algorithm (DTW) to match this sequence with a preset mouth opening amplitude sequence, so as to determine whether the user actually uttered the keyword in the video.

[0205] S4.1 Construction of preset mouth opening amplitude sequence:

[0206] In the embodiment of the present disclosure, before the system is deployed, video clips of natural utterance of "Wo Zhi Dao (I know)" from different users and in different scenarios are collected. Each segment of video extracts the MOAR sequence through steps S1 to S3, and uniformly performs the following preprocessing: uniform length: all template sequences are uniformly adjusted to a fixed length (e.g., 100 frames) using linear interpolation.

[0207] In the embodiment of the present disclosure, based on the above adjustment, a preset mouth opening amplitude sequence set is formed: T = {T (1) , T​(2) ,...,T (N)}, where each T (i) This represents the MOAR sequence template used by a standard user when saying "I know".

[0208] S4.2DTW matching process:

[0209] In this embodiment of the present disclosure, the target mouth opening amplitude sequence Q obtained in step S3 is compared with each T in the preset mouth opening amplitude sequence T. (i) Perform DTW matching.

[0210] For example, let: Q = [q1, q2, ..., q M ], T (i) =[t1,t2,...,t N The recursive formula for the DTW distance matrix is ​​as follows:

[0211]

[0212] The initial boundary condition is D(0,0), and all other conditions are initialized to infinity. The final D(M,N) is the input sequence Q and the template T. (i) The minimum cumulative matching distance.

[0213] S4.3 Matching Decision:

[0214] In this embodiment of the disclosure, the minimum matching distance is determined as follows: D min =min i D(Q,T (i) If the minimum distance D min If the value is less than or equal to θ, then the user in the current video has expressed the keyword "I know," and the match is successful. This matching mechanism can effectively tolerate changes in the speaker's speaking speed and differences in lip movements, and has good temporal alignment capabilities and robust recognition.

[0215] S5: Matching result determination and output.

[0216] After completing the dynamic time warping matching between the target mouth opening amplitude sequence and the preset mouth opening amplitude sequence, the embodiments of this disclosure can further perform reasonableness judgment and output processing on the matching results to ensure that the system has good stability and interpretability in actual business.

[0217] S5.1 Multi-template Fault Tolerance Mechanism:

[0218] Because different users may have different pronunciation rhythms and lip opening amplitudes when expressing the keyword "I know," the system was designed with multiple preset mouth opening amplitude sequences for "I know." For each video to be tested, its mouth opening amplitude MOAR sequence Q will be compared with all the modulo-T sequences. (1) ,T(2) ,...,T (N) Perform DTW matching and calculate the matching distance: D min =min i D(Q,T (i) If there exists any distance value that satisfies: D min If the value is less than or equal to θ, it is determined that the current user has indeed expressed "I know" in the video, and the match is successful. This mechanism can effectively tolerate individual differences such as speaker speed and slight differences in lip movements, thus improving the overall robustness of the system.

[0219] Structured output of S5.2 matching results:

[0220] Once a match is successful, the system will output the following result items in a structured format:

[0221] Target speech keyword content: "I know".

[0222] Video time interval: Frame_start = 215, Frame_end = 305.

[0223] Match score: Based on the minimum DTW distance D*, the confidence score is output after normalization (e.g., between 0 and 1).

[0224] MOAR Feature Summary: Outputs the maximum value, average value, coefficient of variation, and other indicators of MOAR within the given time interval.

[0225] Business trigger signal: If the match is successful, the "User confirmed" flag can be triggered for authorization or compliance confirmation.

[0226] S5.3 Anomaly and Rejection Mechanism:

[0227] To improve system robustness, an anomaly detection module is added during the matching process. If any of the following conditions exist in the MOAR sequence within the matching frame range (215-305), the match will be rejected directly:

[0228] A consistently zero MOAR sequence indicates that the mouth is not open (possibly due to occlusion, detection failure, etc.).

[0229] The MOAR sequence shows very little fluctuation and is approximately horizontal, indicating that mouth movements are not significant.

[0230] The auxiliary judgment mechanism triggers anomalies, such as points 66 / 67 / 68 being closer to the bridge of the nose than points 62 / 63 / 64 (refer to the closed-mouth detection in Step 2).

[0231] The system will output a "rejection" flag in these scenarios, along with an explanation of the reason for the exception, for subsequent manual or system processing.

[0232] S5.4 Interface and Business Integration Capabilities:

[0233] This system supports returning visual lip-reading recognition results to upper-layer business systems via an interface, and can be widely used in: Bank remote authorization systems: verifying whether users clearly say voice keywords such as "I know" or "I agree" in video. Intelligent customer service video compliance platforms: determining whether customers clearly confirm agreement terms in video. Multimodal voice interaction systems: assisting ASR voice recognition modules to improve overall command confirmation rates. The system output supports JSON, XML, and other standard formats, facilitating rapid deployment in various scenarios such as finance, law enforcement, healthcare, and security. In summary, the S5 embodiment fully describes how to transform MOAR matching results into usable decision signals and output structures, ensuring the system has good practicality, scalability, and application value.

[0234] In this embodiment, by introducing facial key point analysis, MOAR mouth opening amplitude function modeling, speech synchronization alignment, and DTW sequence matching technology, the following effects can be achieved: 1. Improve the mouth opening measurement method and enhance discrimination accuracy. Compared with traditional EAR (Eye Length Ratio) or simple key point distance calculation methods, MOAR introduces the concept of area ratio, using multiple key points to form a stable polygon, improving the stability, continuity, and anti-interference ability of mouth opening amplitude estimation. The ability to avoid misjudgment of "lip pursing" is further enhanced by introducing auxiliary structural judgment (such as nasal bridge distance comparison). 2. Combine speech recognition and temporal extension to locate key mouth movement regions. Whisper and other speech recognition models are used to obtain keyword timestamps, and the window is expanded before and after to ensure complete capture of the start and end of mouth shape changes during the user's pronunciation process, avoiding the risk of mismatch caused by truncating mouth shape fragments. 3. Introduce DTW matching to improve robustness and adaptability. The dynamic time warping algorithm can automatically align MOAR sequences of different lengths and rhythms, adapting to individual differences such as varying speaking speeds and mouth opening amplitudes, greatly improving the system's generalization ability. A multi-template fault-tolerance mechanism further improves the system's matching success rate under diverse population conditions. 4. Structured output and rejection mechanism enhance system usability. The system not only outputs matching judgment results but also provides detailed confidence scores, time windows, MOAR statistical features, etc., providing more interpretable evidence for upper-level business systems. The designed abnormal rejection strategies (such as continuous lip-closing and minimal MOAR fluctuation) ensure that the system will not output incorrectly when key lip shapes are missing or detection fails, enhancing security. 5. Highly deployable and with scalable interfaces. The system can be deployed in scenarios such as bank authorization confirmation, remote video contract signing, and multimodal voice security authentication. The standardized output format (JSON / XML) allows for seamless integration into commonly used business systems.

[0235] Based on the same concept, this disclosure also provides an apparatus 100 for determining speech keywords in a video.

[0236] It is understood that the device 100 for determining speech keywords in a video provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0237] Figure 14 This is a block diagram illustrating an apparatus 100 for determining speech keywords in a video according to an exemplary embodiment. (Refer to...) Figure 14 The device includes an acquisition unit 101, an identification unit 102, a determination unit 103, a processing unit 104, and a matching unit 105.

[0238] The acquisition unit 101 is used to acquire a video, which includes human voice and the facial image of the user who made the human voice.

[0239] The recognition unit 102 is used to recognize the key points of the lips in the face images included in each video frame within the video.

[0240] The determining unit 103 is used to perform speech recognition on human voices in the video, and in response to the recognition of target speech keywords, to determine the start timestamp and end timestamp corresponding to the target speech keywords;

[0241] Processing unit 104 is used to obtain the target mouth opening amplitude sequence of the user in the video based on lip key points, start timestamp and end timestamp, wherein the mouth opening amplitude sequence corresponds to different speech keywords;

[0242] The matching unit 105 matches the target mouth opening amplitude sequence with the preset mouth opening amplitude sequence corresponding to the target speech keyword. If the target mouth opening amplitude sequence matches the preset mouth opening amplitude sequence, it is determined that the user's voice contains the target speech keyword.

[0243] In one embodiment, based on lip key points, a start timestamp, and an end timestamp, the processing unit 104 obtains the target mouth opening amplitude sequence of the user in the video in the following manner: determining the mouth opening amplitude based on lip key points to obtain the user's mouth opening amplitude sequence, wherein the mouth opening amplitude is used to reflect the opening and closing state of the user's lips; and extracting the target mouth opening amplitude sequence that generates the target speech keyword from the mouth opening amplitude sequence based on the start timestamp and the end timestamp.

[0244] In one embodiment, based on lip key points, start timestamp, and end timestamp, the processing unit 104 obtains the user's target mouth opening amplitude sequence in the video in the following manner: based on the start timestamp and end timestamp, a video segment that generates the target speech keyword is extracted from the video; lip key points of each video frame within the video segment are identified; and based on the lip key points, the user's target mouth opening amplitude sequence in the video is determined.

[0245] In one embodiment, the mouth opening amplitude is determined as follows: all external lip key points and all internal lip key points included in the lip key points are determined; a first polygon formed by all external lip key points is determined, and a first area of ​​the first polygon is determined; a second polygon formed by all internal lip key points is determined, and a second area of ​​the second polygon is determined; the ratio of the second area to the first area is determined as the mouth opening amplitude.

[0246] In one embodiment, the determining unit 103 determines the mouth opening amplitude based on lip key points in the following manner: determining all auxiliary key points and acquiring some inner lip key points, including the left inner lip key point, the middle inner lip key point, and the right inner lip key point; based on all auxiliary key points, the left inner lip key point, the middle inner lip key point, and the right inner lip key point, obtaining multiple sets of distances, including the first left distance and the second left distance between each auxiliary key point and the left inner lip key point, the first middle distance and the second middle distance between each auxiliary key point and the middle inner lip key point, and the first right distance and the second right distance between each auxiliary key point and the right inner lip key point; if any of the multiple sets of distances does not meet the distance condition, the mouth opening amplitude is set to 0.

[0247] In one embodiment, the matching unit 105 matches the target mouth opening amplitude sequence with the preset mouth opening amplitude sequence corresponding to the target speech keyword in the following manner: DTW distance is calculated for each target mouth opening amplitude sequence and the preset mouth opening amplitude sequence corresponding to the target speech keyword, and the minimum DTW distance is determined from the calculated DTW distances to obtain a minimum DTW distance set; the minimum DTW distance in the minimum DTW distance set is determined; if the minimum DTW distance is less than or equal to a distance threshold, it is determined that the user's voice in the video includes the target speech keyword; if the minimum DTW distance is greater than the distance threshold, it is determined that the user's voice in the video does not include the target speech keyword.

[0248] In one embodiment, the determining unit 103 determines the start and end timestamps corresponding to the target speech keywords in the following manner: the video is recognized to obtain a time information sequence, which includes the start and end timestamps of the text content in the video; if the text content includes the target speech keywords, the initial start and end timestamps of the target speech keywords are determined; the N frames before the initial start timestamp are determined as the start timestamps, and the M frames after the initial end timestamps are determined as the end timestamps, where N and M are positive integers.

[0249] In one implementation, the video is acquired in a financial business scenario.

[0250] In one embodiment, after determining that the user's voice in the video includes the target speech keyword, the determining unit 103 is further configured to: output at least one of the following: the target speech keyword; the start timestamp and the end timestamp; the target mouth opening amplitude sequence; and the business action, including whether business logic is triggered.

[0251] In one embodiment, the matching unit 105 is further configured to: if the target mouth opening amplitude sequence is determined to be invalid within the target time period of the start timestamp and the end timestamp, then the target mouth opening amplitude sequence and the preset mouth opening amplitude sequence corresponding to the target speech keyword are not matched; the target mouth opening amplitude sequence being invalid includes at least one of the following situations: the mouth opening amplitude is continuously 0; the mouth opening amplitude is continuously less than the amplitude threshold; any one of the multiple distances does not meet the distance condition.

[0252] In one implementation, the preset mouth opening amplitude sequence is pre-trained as follows: video clips of different users expressing target speech keywords in different scenarios are collected; the mouth opening amplitude sequence in the video clips is extracted to obtain the mouth opening amplitude sequence corresponding to different users in different scenarios; the length of the mouth opening amplitude sequence corresponding to different users in different scenarios is unified to obtain the preset mouth opening amplitude sequence corresponding to the target speech keyword.

[0253] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0254] Figure 15 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure. Figure 15As shown, one embodiment of this disclosure provides an electronic device 200. The electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. The memory 201 stores instructions. The processor 202 executes the method for determining speech keywords in a video according to embodiments of this disclosure by calling the instructions stored in the memory 201. The processor 202 is connected to both the memory 201 and the I / O interface 203, for example, via a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including a program for an essay correction method according to embodiments of this disclosure. The processor 202 executes various functional applications and data processing of the electronic device 200 by running the programs stored in the memory 201.

[0255] In this embodiment of the disclosure, the processor 202 may be implemented in at least one of the following hardware forms: digital signal processor (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 202 may be one or a combination of several of the following: central processing unit (CPU) or other processing units with data processing capability and / or instruction execution capability.

[0256] The memory 201 in this embodiment may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0257] In this embodiment of the disclosure, the I / O interface 203 can be used to receive input instructions (such as numeric or character information, and to generate key signal inputs related to user settings and function control of the electronic device 200), and can also output various information (such as images or sounds) to the outside. In this embodiment of the disclosure, the I / O interface 203 may include one or more of the following: a physical keyboard, function keys (such as volume control keys, power buttons, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0258] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by the processor 202 of the electronic device 200 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0259] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0260] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of the electronic device 200, the electronic device 200 is able to perform the methods described above. This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method for determining speech keywords in a video as described in any of the foregoing embodiments, or implements the function of the device for determining speech keywords in a video as described in any of the foregoing embodiments.

[0261] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0262] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0263] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0264] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0265] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0266] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0267] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for determining speech keywords in a video, characterized in that, include: Acquire a video, the video including human voice and the facial image of the user corresponding to the human voice; Identify the key lip points of the facial images included in each video frame within the video; In the video, speech recognition is performed on the human voice, and in response to the recognition of target speech keywords, the start timestamp and end timestamp corresponding to the target speech keywords are determined; Based on the lip key points, the start timestamp, and the end timestamp, the target mouth opening amplitude sequence of the user in the video is obtained, wherein the mouth opening amplitude sequence corresponds to different voice keywords; The target mouth opening amplitude sequence is matched with the preset mouth opening amplitude sequence corresponding to the target voice keyword. If the target mouth opening amplitude sequence matches the preset mouth opening amplitude sequence, it is determined that the voice emitted by the user includes the target voice keyword. The step of obtaining the target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp, and the end timestamp includes: The mouth opening amplitude is determined based on the key points of the lips to obtain the user's mouth opening amplitude sequence, wherein the mouth opening amplitude is used to reflect the user's lip opening and closing state; Based on the start timestamp and the end timestamp, extract the target mouth opening amplitude sequence that generates the target speech keyword from the mouth opening amplitude sequence; Determining the mouth opening range based on the key lip points includes: All auxiliary key points are determined, and some inner lip key points are obtained, including the left inner lip key point, the middle inner lip key point, and the right inner lip key point. Based on all the auxiliary key points, the left inner lip key point, the middle inner lip key point, and the right inner lip key point, multiple sets of distances are obtained. These multiple sets of distances include the first left distance and the second left distance between all the auxiliary key points and the left inner lip key point, the first middle distance and the second middle distance between all the auxiliary key points and the middle inner lip key point, and the first right distance and the second right distance between all the auxiliary key points and the right inner lip key point. If any of the multiple distance sets does not meet the distance condition, the mouth opening amplitude will be set to 0.

2. The method according to claim 1, characterized in that, The process of obtaining the target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp, and the end timestamp includes: Based on the start timestamp and the end timestamp, a video segment that generates the target speech keywords is extracted from the video. Identify the key lip points in each video frame within the video segment; Based on the key lip points, the target mouth opening amplitude sequence of the user in the video is determined.

3. The method according to claim 1, characterized in that, The mouth opening range is determined in the following way: Identify all the key points of the outer lip and all the key points of the inner lip included in the key points of the lip; Determine the first polygon formed by all the key points of the outer lip, and determine the first area of ​​the first polygon; Determine the second polygon formed by all the key points of the inner lip, and determine the second area of ​​the second polygon; The ratio of the second area to the first area is determined as the mouth opening amplitude.

4. The method according to claim 1, characterized in that, The step of matching the target mouth opening amplitude sequence with the preset mouth opening amplitude sequence corresponding to the target speech keyword includes: The target mouth opening amplitude sequence and the preset mouth opening amplitude sequence corresponding to the target speech keyword are subjected to dynamic time warping distance calculation one by one, and the minimum dynamic time warping distance is determined in the calculated dynamic time warping distance to obtain the minimum dynamic time warping distance set. Determine the minimum dynamic time warp distance in the set of minimum dynamic time warp distances; If the minimum dynamic time warp distance is less than or equal to the distance threshold, then it is determined that the target speech keyword is included in the human voice emitted by the user in the video; If the minimum dynamic time warp distance is greater than the distance threshold, then it is determined that the target speech keyword is not included in the human voice emitted by the user in the video.

5. The method according to claim 1, characterized in that, Determining the start and end timestamps corresponding to the target speech keywords includes: The video is identified to obtain a time information sequence, which includes the start and end timestamps of the text content within the video. If the text content includes the target speech keyword, then the initial start timestamp and initial end timestamp of the target speech keyword are determined. The N frames before the initial start timestamp are determined as the start timestamp, and the M frames after the initial end timestamp are determined as the end timestamp, where N and M are positive integers.

6. An apparatus for determining speech keywords in a video, characterized in that, include: The acquisition unit is used to acquire video, the video including human voice and the facial image of the user corresponding to the human voice; The recognition unit is used to identify the key points of the lips in the face images included in each video frame of the video. The determining unit is configured to perform speech recognition on the human voice in the video, and in response to the recognition of a target speech keyword, determine the start timestamp and end timestamp corresponding to the target speech keyword; The processing unit is configured to obtain the target mouth opening amplitude sequence of the user in the video based on the lip key points, the start timestamp, and the end timestamp, wherein the mouth opening amplitude sequence corresponds to different speech keywords; The matching unit matches the target mouth opening amplitude sequence with the preset mouth opening amplitude sequence corresponding to the target speech keyword. If the target mouth opening amplitude sequence matches the preset mouth opening amplitude sequence, it is determined that the voice emitted by the user includes the target speech keyword. Specifically, based on the lip key points, the start timestamp, and the end timestamp, the processing unit obtains the target mouth opening amplitude sequence of the user in the video in the following manner: determining the mouth opening amplitude based on the lip key points to obtain the user's mouth opening amplitude sequence, wherein the mouth opening amplitude is used to reflect the opening and closing state of the user's lips; and extracting the target mouth opening amplitude sequence that generates the target speech keyword from the mouth opening amplitude sequence based on the start timestamp and the end timestamp. The determining unit determines the mouth opening amplitude based on the lip key points in the following manner: It determines all auxiliary key points and acquires a portion of the inner lip key points, including the left inner lip key point, the middle inner lip key point, and the right inner lip key point; based on all auxiliary key points, the left inner lip key point, the middle inner lip key point, and the right inner lip key point, it obtains multiple sets of distances, including a first left distance and a second left distance between all auxiliary key points and the left inner lip key point, a first middle distance and a second middle distance between all auxiliary key points and the middle inner lip key point, and a first right distance and a second right distance between all auxiliary key points and the right inner lip key point; if any of the multiple sets of distances does not meet the distance condition, the mouth opening amplitude is set to 0.

7. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions that can be executed by a processor; The processor is configured to execute the computer program or instructions to implement the steps of the method for determining speech keywords in the video according to any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method for determining speech keywords in a video according to any one of claims 1 to 5.

Images