A live person recognition method and related apparatus

Abstract

The application discloses a live broadcast recognition method and related device, and relates to the field of data recognition. Video content collection and audio content collection are performed on obtained live broadcast content, and video frame data and audio frame data are obtained. Mouth shape feature indexes are extracted from the video frame data, and first lip pronunciation feature data are obtained. Speech activity detection and speech speed feature index extraction are performed on the audio frame data, and second lip pronunciation feature data and speech speed feature data are obtained. Based on the first and second pronunciation feature data, a synchronous matching rate between a pronunciation mouth shape and real speech is determined. Based on the synchronous matching rate and the speech speed feature data, a live broadcast recognition result is obtained. The video mouth shape pronunciation feature and the audio real pronunciation feature are combined, the user speech speed feature meets the live broadcast sound interval, and the sound frame and the mouth pronunciation feature both meet the synchronous rate requirement, so that the live broadcast is determined, and the judgment accuracy is higher.

Description

Technical Field

[0001] This application relates to the field of data recognition technology, and in particular to a live-streaming recognition method and related apparatus. Background Technology

[0002] With the rapid development of the online live streaming industry, more and more businesses are using video platforms for product sales, knowledge dissemination, and entertainment. To save manpower and improve efficiency, some organizations or individuals are attempting to use technologies such as pre-recorded videos, face-swapping, and voice synthesis to simulate live streams, thereby bypassing platform review systems and achieving goals such as "fake" sales and "automated" live streaming. These practices not only undermine users' trust in the platform's authenticity but also introduce numerous risk control and regulatory challenges. Therefore, identifying such risky behaviors to improve the authenticity of live streams has become an urgent problem to solve. Summary of the Invention

[0003] In view of the above problems, this application provides a method and related apparatus for live-streaming identification to improve the accuracy of live-streaming identification. The specific solution is as follows:

[0004] The first aspect of this application provides a method for identifying live streamers, including:

[0005] The acquired live content is subjected to video content acquisition and audio content acquisition respectively, resulting in video frame data and audio frame data;

[0006] Mouth shape feature indicators are extracted from the video frame data to obtain the first pronunciation feature data of the lips;

[0007] Speech activity detection and speech rate feature extraction are performed on the audio frame data to obtain the second pronunciation feature data of the lips and speech rate feature data;

[0008] Based on the first pronunciation feature data and the second pronunciation feature data, the synchronization matching rate between the pronunciation mouth shape and the actual speech is determined;

[0009] Based on the synchronization matching rate and the speech rate feature data, the live stream recognition result is obtained.

[0010] In one possible implementation, the step of extracting lip shape feature indicators from the video frame data to obtain first pronunciation feature data of the lips includes:

[0011] For each video frame, the mouth opening and closing index, the upper and lower lip area change index, and the mouth symmetry index are extracted respectively.

[0012] For each video frame, the mouth opening and closing index data, upper and lower lip area change characteristic index data, and mouth symmetry characteristic index data of the video frame are fused according to a preset weight to obtain the index fusion data of the video frame.

[0013] Based on the fusion data of the aforementioned indicators and the lip shape threshold, it is determined whether the lip shape in the video frame is a vocal lip shape.

[0014] In one possible implementation, the step of performing speech activity detection and speech rate feature extraction on the audio frame data to obtain the second articulation feature data of the lips and speech rate feature data includes:

[0015] Speech activity detection is performed on each audio frame based on a speech activity detection algorithm to determine whether there is real speech in each audio frame.

[0016] The speech rate feature data is determined based on the number of target audio frames contained in multiple time windows and the duration of the multiple time windows. The target audio frames are the audio frames in the multiple time windows in which the real speech exists.

[0017] In one possible implementation, determining the speech rate feature data based on the number of target audio frames contained in multiple time windows and the duration of the multiple time windows includes:

[0018] For each time window, the speech rate value is determined based on the number of target audio frames contained within the time window and the duration of the time window.

[0019] The speech rate standard deviation is determined based on the speech rate value for each time window.

[0020] In one possible implementation, determining the synchronization matching rate between the lip shape and the actual speech based on the first and second pronunciation feature data includes:

[0021] The synchronization matching rate is determined based on the number of audio-visual synchronizations and the number of audio frames identified as real speech. The audio-visual synchronization is defined as the mouth shape in the video frame at the same acquisition time being the mouth shape for pronunciation and the presence of real speech in the audio frame.

[0022] In one possible implementation, obtaining the live stream identification result based on the synchronization matching rate and the speech rate feature data includes:

[0023] If the synchronization matching rate is greater than the synchronization matching threshold, the speech rate is within the speed range, and the speech rate standard deviation is not less than the standard deviation threshold, then it is determined to be a live broadcast by a real person.

[0024] In one possible implementation, obtaining the live stream recognition result based on the synchronization matching rate and the speech rate feature data further includes:

[0025] If the synchronization matching rate is not greater than the synchronization matching threshold, or the speech rate value is not within the speed value range, or the speech rate standard deviation is less than the standard deviation threshold, then corresponding risk management measures shall be taken according to the degree of deviation of the synchronization matching rate, or the degree of deviation of the speech rate value, or the degree of deviation of the speech rate standard deviation.

[0026] A second aspect of this application provides a live-streaming recognition device, comprising:

[0027] The frame data acquisition module is used to acquire video content and audio content from the obtained live content, and obtain video frame data and audio frame data.

[0028] The video frame processing module is used to extract mouth shape feature indicators from the video frame data to obtain the first pronunciation feature data of the lips;

[0029] An audio frame processing module is used to perform speech activity detection and speech rate feature extraction on the audio frame data to obtain the second pronunciation feature data of the lips and speech rate feature data.

[0030] The frame data matching module is used to determine the synchronization matching rate between the mouth shape and the actual speech based on the first pronunciation feature data and the second pronunciation feature data; and,

[0031] The live-streaming recognition module is used to obtain the live-streaming recognition result based on the synchronization matching rate and the speech rate feature data.

[0032] A third aspect of this application provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement the live-streaming recognition method of the first aspect or any implementation thereof.

[0033] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0034] The memory is used to store computer programs;

[0035] The processor is used to execute the computer program so that the electronic device can implement the live-streaming recognition method of the first aspect or any implementation thereof.

[0036] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the live-streaming recognition method of the first aspect or any implementation thereof.

[0037] By employing the above technical solution, the live-stream recognition method provided in this application acquires video and audio content from the live-stream content, obtaining video frame data and audio frame data. Mouth shape feature indicators are extracted from the video frame data to obtain first lip pronunciation feature data. Speech activity detection and speech rate feature indicators are extracted from the audio frame data to obtain second lip pronunciation feature data and speech rate feature data. Based on the first and second pronunciation feature data, the synchronization matching rate between the mouth shape and the actual speech is determined. Based on the synchronization matching rate and speech rate feature data, the live-stream recognition result is obtained. This method combines video mouth shape pronunciation features with audio actual pronunciation features. When the user's speech rate features match the range of a real person's voice, and both the voice frame and mouth pronunciation features meet the synchronization rate requirement, it is proven to be a live-stream from a real person, resulting in a higher accuracy rate for judging live-streams from real people. Attached Figure Description

[0038] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0039] Figure 1 An architecture diagram of a live-streaming recognition system provided in this application;

[0040] Figure 2 A flowchart of a live-streaming identification method provided in this application;

[0041] Figure 3 A structural diagram of a live-streaming recognition device provided in this application;

[0042] Figure 4 This is a structural diagram of an electronic device provided in this application. Detailed Implementation

[0043] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0044] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0045] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0046] See Figure 1 , Figure 1 A schematic diagram of the architecture of a live-streaming recognition system is shown. The system may include a terminal 100 and a server 200. The server 200 can provide the live-streaming recognition method provided in this embodiment to one or more terminals.

[0047] The terminal 100 may have a live-streaming recognition application installed. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the live-streaming recognition interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0048] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0049] The following description Figure 1 The product form of the mid-terminal 100;

[0050] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, in-vehicle device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0051] Terminal 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the terminal or multifunctional device; it may include more or fewer components, or a combination of certain components, or different components.

[0052] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0053] Among them, the input device can receive input data, etc.

[0054] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display the interface for live-stream recognition, processing results, etc.

[0055] The memory can be used to store software code related to the live-stream recognition method, the processor can execute the steps of the live-stream recognition method, and can also schedule other units (such as the above-mentioned input unit and display unit) to achieve the corresponding functions.

[0056] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.

[0057] In this embodiment of the application, the radio frequency unit can send data to the server 200 and receive the processing results sent by the server 200.

[0058] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.

[0059] Terminal 100 also includes a power source (such as a battery) for supplying power to the various components.

[0060] Terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0061] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.

[0062] The memory can be used to store software code related to the live-streaming recognition method, the processor can execute the steps of the live-streaming recognition method of the chip, and can also schedule other units to achieve corresponding functions.

[0063] Currently, common methods for identifying live streams featuring real people include content recognition, watermark verification, and device fingerprinting. However, these methods have certain blind spots when it comes to identifying fake live videos. For example, using a combination of TTS (Text-to-Speech) and pre-recorded live video can often bypass traditional image or audio analysis mechanisms. Therefore, current identification methods still have limitations in identifying live streams featuring real people.

[0064] To address the aforementioned problems, this application provides a method for identifying live streamers. The live streamer identification method of this application will be described in detail below with reference to the accompanying drawings.

[0065] Reference Figure 2 , Figure 2 This application provides a flowchart illustrating a live-stream recognition method. Figure 2 As shown in the figure, the live streaming identification method provided in this application embodiment may include steps 201 to 205, which are described in detail below.

[0066] 201. Collect video content and audio content from the acquired live content to obtain video frame data and audio frame data.

[0067] Specifically, when capturing relevant live content, the live content (video stream and audio stream) can be obtained from the live address using capture tools. For example, FFmpeg (an open-source computer program that can record, convert, and stream digital audio and video) can be used to capture video and audio streams in real time from the broadcaster's RTMP (Real Time Messaging Protocol) live address.

[0068] For the video portion, a frame rate of 25fps can be used to capture continuous video frames, ensuring timestamp synchronization. For the audio portion, an audio stream can be captured at 16kHz to obtain audio segments. To achieve alignment and synchronization between video and audio frames, the audio portion is also divided into 40ms audio frames.

[0069] The frame rate used here is based on the default video stream configuration of current mainstream live streaming platforms, and this frame rate also has good temporal resolution in lip-sync detection. Similarly, the sampled audio uses a 16kHz mono encoding format, which is a widely accepted standard configuration in subsequent speech activity detection and lip-sync analysis. This ensures the integrity of the effective speech frequency bands, simplifies the processing flow, and ensures efficient and stable system operation.

[0070] It is understood that those skilled in the art can adjust and select the above-mentioned acquisition methods and corresponding acquisition parameters as needed, which will not be elaborated further here.

[0071] 202. Extract mouth shape feature indicators from video frame data to obtain the first pronunciation feature data of the lips.

[0072] Specifically, the RetinaFace algorithm can be used to locate the face region for each video frame. Within the detected face region, the Dlib 68-point keypoint model is used to locate facial keypoints.

[0073] For each video frame, the following indices were extracted: mouth opening degree, upper and lower lip area change, and mouth symmetry. Specifically, the vertical distance between the upper and lower lips was calculated by selecting key points on the lips and used as the mouth opening degree index.

[0074] The contour area of ​​the lip closure region can be used to reflect the dynamic changes in the degree of mouth opening, serving as a characteristic indicator of changes in the area of ​​the upper and lower lips.

[0075] Because the symmetry of mouth shape changes is high in normal speech, unnatural asymmetry is easily seen in synthetic animation or pre-recorded videos. The difference in the changes of key points of mouth shape on the left and right sides can be calculated to identify fake mouth shape or unnatural mouth movements, thus obtaining a mouth shape symmetry characteristic index.

[0076] Then, for each video frame, the mouth opening and closing index data, upper and lower lip area change characteristic index data, and mouth symmetry characteristic index data of the video frame are fused according to the pre-set weights to obtain the index fusion data of the video frame.

[0077] Based on this metric, the data is fused with a mouth shape threshold to determine whether the mouth shape in a video frame is a vocalization mouth shape. Then, the video frames with vocalization mouth shapes are identified, yielding the first vocalization feature data.

[0078] 203. Perform speech activity detection and speech rate feature extraction on the audio frame data to obtain the second pronunciation feature data of the lips and speech rate feature data.

[0079] Specifically, speech activity detection algorithms, such as WebRTC VAD (Voice Activity Detection), can be used to detect speech activity in each audio frame, determine whether there is real speech in each audio frame, and obtain second pronunciation feature data.

[0080] Speech rate characteristic data refers to the number of active frames of a user's speech per unit of time, used to evaluate whether the speech output is natural. In normal speech, a person's speech rate naturally fluctuates due to emotions, semantic structure, and thought processes. However, TTS-synthesized audio or pre-recorded materials may exhibit anomalies such as "speech rate too slow to prevent lip-syncing errors" or "uniform speech rate."

[0081] To achieve accurate speech rate determination, speech rate feature data can be composed of average speech rate and speech rate variance. The average speech rate is determined based on the number of target audio frames contained within a time window and the duration of multiple time windows. Target audio frames are the audio frames containing actual speech within the time window. The standard deviation of the speech rate is determined based on the speech rate value for each time window.

[0082] 204. Based on the first and second pronunciation feature data, determine the synchronization matching rate between the pronunciation mouth shape and the actual speech.

[0083] Specifically, the synchronization matching rate can be determined based on the number of audio-visual synchronizations and the number of audio frames identified as real speech. Audio-visual synchronization means that the mouth shape in the video frame at the same acquisition time is the mouth shape for pronunciation and that real speech exists in the audio frame.

[0084] 205. Based on the synchronization matching rate and speech rate feature data, the results of live-streaming identification are obtained.

[0085] Specifically, to ensure the reliability of the live stream recognition results, a live stream is identified as live only if the synchronization matching rate is greater than the synchronization matching threshold, the speech rate is within the specified range, and the speech rate standard deviation is not less than the standard deviation threshold. In other words, a live stream is only recognized as live if all of the above conditions are met. In other cases, corresponding risk management measures may be taken based on the degree of deviation of the synchronization matching rate, the speech rate, or the speech rate standard deviation.

[0086] This live-stream voice recognition method extracts pronunciation feature data from video and audio frames separately. It uses a speech rate that matches the pronunciation range of a real person, and the video and audio frames meet the requirements for synchronized speech. This method can meet the speech feature recognition needs of various types of clients and servers, and has a higher recognition accuracy.

[0087] As mentioned above, in step 202, mouth shape feature indicators are extracted from video frame data to obtain the first pronunciation feature data of the lips. Specifically, the extraction of mouth shape feature indicators may include:

[0088] The RetinaFace algorithm is used to locate the face region. Within the detected face region, the Dlib 68-point keypoint model is used to locate facial keypoints.

[0089] Mouth opening and closing index:

[0090] Extract key points of the mouth (numbered 49 to 68), focusing on calculating the vertical distance between the upper and lower lips as an indicator of mouth opening degree, and calculate the mouth opening degree D(t) for each frame:

[0091] D(t) = |y66-y62|;

[0092] Where y66 represents the ordinate of the mouth keypoint 66, and y62 represents the ordinate of the mouth keypoint 62. A continuous sequence of changes over time, i.e., the D(t) characteristic curve, can be obtained from D(t) of each frame.

[0093] Characteristic indicators of upper and lower lip area changes:

[0094] Calculating the area change of the lip opening region reflects the dynamics of the degree of mouth opening, suitable for describing subtle fluctuations in the transition between "opening" and "closing" the mouth, and also suitable for rhythm analysis. The contour area of ​​the lip closing region can be used, based on the outer lip contour points in the 68 facial keypoints: point number 48 → 59 (clockwise).

[0095] ;

[0096] Where: n=12 corresponds to the number of points on the outer lip contour, x i and y i These represent the horizontal and vertical coordinates, respectively.

[0097] This formula is the Shoelace Formula for polygon area, which is often used in the field of image processing for calculating contour area.

[0098] Mouth shape symmetry characteristic indicators:

[0099] This indicator reflects whether the two sides of the mouth open and close symmetrically, and is used to identify fake mouth shapes or unnatural mouth movements. It mainly calculates the difference in changes of key points of the mouth shape between the left and right sides. Normal speech has high symmetry in mouth shape changes, while synthetic animation or pre-recorded videos are prone to unnatural asymmetry.

[0100] S(t)=|x(60)-x(64)|-|x(48)-x(54)|;

[0101] Where x(60) represents the x-coordinate of the key point 60 of the mouth, x(64) represents the x-coordinate of the key point 64 of the mouth, x(48) represents the x-coordinate of the key point 48 of the mouth, and x(54) represents the x-coordinate of the key point 54 of the mouth.

[0102] Then, the above-mentioned feature indicators are standardized:

[0103] ;

[0104] Where μ and σ are the historical mean and standard deviation of the corresponding indicators, respectively.

[0105] Finally, the data is merged to obtain the fused indicator data:

[0106] ;

[0107] Where w represents the combined weight of the three indicators, for example W D =0.6 W A =0.25 W s =0.15.

[0108] It is understood that those skilled in the art can adjust the position of each key point of the mouth and the fusion weight of each indicator as needed, without any restrictions.

[0109] The process of determining the second pronunciation feature data includes:

[0110] The WebRTC VAD algorithm is used to detect speech activity in each frame of audio to obtain the speech activity state of each frame:

[0111] We obtain: V(t) = {1 indicates the presence of speech, 0 indicates the absence of speech};

[0112] Generate a time series V(t) of speech activity and record the speech activity status at each time point.

[0113] Next, the speech rate characteristic indicators are:

[0114] ;

[0115] Where, N speech This represents the number of audio frames (40ms / frame) deemed to contain actual speech within the time window T, where T represents the total time window (typically 5-10 seconds). The specific determination method is shown in Table 1 below:

[0116] Table 1

[0117] Judgment item Judgment Logic Example threshold Speech rate too slow <![CDATA[If R s <R min Average speaking time per second is too short]]> <![CDATA[R min =10 frames / s]]> Constant speaking rate Under multiple sliding time windows, the standard deviation of speech rate σ(Rs) < δ δ=15 frames / second

[0118] A constant speech rate refers to a very small standard deviation, i.e., minimal fluctuation, across multiple consecutive time windows (e.g., 5). For easier comparison, the speech rate can be normalized as follows:

[0119] Rs(t)∈[0,1];

[0120] The specific process for determining the synchronization matching rate between lip movements and actual speech includes:

[0121] ;

[0122] in M(t) represents the number of frames in which mouth shapes represent articulation and where actual articulation occurs in speech, i.e., the number of audio and video frames containing M(t). The number of V(t) = 1. Where M(t) = {1 represents the presence of a mouth shape indicating the existence of a sound, and 0 represents the absence of a mouth shape indicating the absence of a sound}. To determine the number of audio frames that are actual speech.

[0123] To further improve recognition accuracy and reduce false positive rate, a judgment method combining speech rate and synchronization matching rate is adopted:

[0124] (1) Constraints on the reasonableness of speech rate:

[0125] Set the ratio of active speech frames within a continuous time window Tg∈[5,10] seconds, R s When the value is ∈[0.15,0.85], the speech segment is considered to have natural speech rate characteristics, excluding the constant or extremely slow speech rate patterns commonly found in speech synthesis systems.

[0126] (2) Frame-level synchronization rate requirements (local features, determined based on synchronization rate in step 5):

[0127] When R > 0.9, it is determined to be the voice of a live broadcaster, and the risk is low.

[0128] When 0.5 ≤ R ≤ 0.9, it is judged as medium risk, and manual review is recommended;

[0129] When R < 0.5, it is considered high risk, which may be a fake broadcast, and may trigger an automatic warning or ban.

[0130] As a specific application of the aforementioned live-streaming recognition method, if the speech rate characteristics of each interval in a live stream are normal, and if the frame synchronization rate is high, it indicates that the broadcaster is indeed speaking normally, meeting the conditions for a live stream. Conversely, if the speech rate is slow or constant, and the speech signal is very active but the lip-sync matching rate is low, it is suspected to be a "fake speech" or "pre-recorded + speech synthesis" scenario.

[0131] For example: Speech rate feature determination: Data as shown in Table 2 was obtained from audio and video analysis during a live broadcast:

[0132] Table 2

[0133] Time interval (seconds) Number of audio frames containing real speech Number of audio frames contained within the time interval Normalized speech rate Rs 5 75 250 0.3 5 80 250 0.32 5 110 250 0.44 5 60 250 0.24 5 70 250 0.28 5 95 250 0.38

[0134] All these values ​​fall within the range of [0.15, 0.85], indicating that the speaking speed is reasonable and there is no abnormal constant or extremely slow speaking speed.

[0135] For example, as shown in Table 3:

[0136] Table 3

[0137] Time interval (seconds) Number of audio frames containing real speech Normalized speech rate Rs 5 75 0.3 5 75 0.3 5 75 0.3 5 75 0.3 5 75 0.3

[0138] The speech rate was perfectly consistent (0.30) for every 5-second segment, with zero fluctuation. This is likely a characteristic of the constant speech rate output of a TTS system; such precise and stable output is virtually impossible in real human speech.

[0139] For example, as shown in Table 4:

[0140] Table 4

[0141] Time interval (seconds) Number of audio frames containing real speech Normalized speech rate Rs 5 15 0.06 5 20 0.08 5 10 0.04

[0142] The number of audio frames is significantly low, with a normalization value below 0.15, indicating that the broadcaster barely speaks, or that the audio is synthesized through editing with unnatural rhythms.

[0143] Mouth shape feature determination: as shown in Table 5:

[0144] Table 5

[0145] Time frame (t) Mouth shape index fusion data Mouth shape threshold for pronunciation (threshold 0.5) VAD speech activity V(t) 1 0.2 0 0 2 0.6 1 1 3 0.7 1 1 4 0.4 0 1 5 0.1 0 0 6 0.8 1 1 7 0.9 1 0 8 0.3 0 0 9 0.5 1 1 10 0.4 0 1

[0146] Referring to the table above, based on the set threshold:

[0147] If the synchronization rate is >90%, it is considered a real person. If it is <50%, it is very likely a virtual live stream.

[0148] Among them, WebRTC VAD detected 6 audio frames containing real speech.

[0149] Mouth shape analysis revealed that the mouth articulation features of four video frames were greater than 0.5.

[0150] The synchronization matching rate is: Synchronization matching rate = 4 / 6 = 66.7%. This can be classified as medium risk and may require manual review.

[0151] The above describes a live-streaming identification method provided by the embodiments of this application. The following describes the apparatus for performing the above-described live-streaming identification method.

[0152] Please see Figure 3 , Figure 3 This is a schematic diagram of a live-streaming recognition device provided in an embodiment of this application. Figure 3 As shown, the live-streaming recognition device includes:

[0153] The frame data acquisition module 301 is used to acquire video content and audio content from the acquired live content to obtain video frame data and audio frame data.

[0154] The video frame processing module 302 is used to extract mouth shape feature indicators from video frame data to obtain the first pronunciation feature data of the lips;

[0155] The audio frame processing module 303 is used to perform speech activity detection and speech rate feature extraction on the audio frame data to obtain the second pronunciation feature data of the lips and the speech rate feature data.

[0156] Frame data matching module 304 is used to determine the synchronization matching rate between the mouth shape and the actual speech based on the first pronunciation feature data and the second pronunciation feature data; and,

[0157] The live stream recognition module 305 is used to obtain live stream recognition results based on synchronization matching rate and speech rate feature data.

[0158] In one possible implementation, the process by which the video frame processing module 302 extracts lip shape feature indicators from the video frame data to obtain the first articulation feature data of the lips includes:

[0159] For each video frame, the mouth opening and closing index, the upper and lower lip area change index, and the mouth symmetry index are extracted respectively.

[0160] For each video frame, the mouth opening and closing index data, upper and lower lip area change characteristic index data, and mouth symmetry characteristic index data of the video frame are fused according to the pre-set weights to obtain the index fusion data of the video frame.

[0161] Based on the fusion of indicator data and the lip shape threshold, it is determined whether the lip shape in the video frame is a vocal lip shape.

[0162] In one possible implementation, the audio frame processing module 303 performs speech activity detection and speech rate feature extraction on the audio frame data to obtain second lip articulation feature data and speech rate feature data, including:

[0163] Speech activity detection is performed on each audio frame based on a speech activity detection algorithm to determine whether there is real speech in each audio frame.

[0164] Based on the number of target audio frames contained in multiple time windows and the duration of the multiple time windows, speech rate feature data is determined. The target audio frames are audio frames containing real speech in multiple time windows.

[0165] In one possible implementation, the audio frame processing module 303 determines the speech rate feature data based on the number of target audio frames contained in multiple time windows and the duration of the multiple time windows, including:

[0166] For each time window, the speech rate value is determined based on the number of target audio frames contained within the time window and the duration of the time window.

[0167] The standard deviation of speech rate is determined based on the speech rate value for each time window.

[0168] In one possible implementation, the process by which the frame data matching module 304 determines the synchronization matching rate between the articulated mouth shape and the actual speech based on the first and second articulation feature data includes:

[0169] Based on the number of audio-visual synchronizations and the number of audio frames identified as real speech, the synchronization matching rate is determined. Audio-visual synchronization is defined as the mouth shape in the video frame at the same acquisition time being the mouth shape for pronunciation and the presence of real speech in the audio frame.

[0170] In one possible implementation, the process by which the live-stream recognition module 305 obtains the live-stream recognition result based on synchronization matching rate and speech rate feature data includes:

[0171] If the synchronization matching rate is greater than the synchronization matching threshold, the speech rate is within the speed range, and the speech rate standard deviation is not less than the standard deviation threshold, then it is determined to be a live broadcast by a real person.

[0172] In one possible implementation, the process by which the live-stream recognition module 305 obtains the live-stream recognition result based on synchronization matching rate and speech rate feature data also includes:

[0173] If the synchronization matching rate is not greater than the synchronization matching threshold, or the speech rate is not within the speed range, or the speech rate standard deviation is less than the standard deviation threshold, then corresponding risk management measures shall be taken according to the degree of deviation of the synchronization matching rate, or the degree of deviation of the speech rate, or the degree of deviation of the speech rate standard deviation.

[0174] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0175] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0176] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0177] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the live-streaming recognition methods provided in this application.

[0178] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the live-streaming recognition methods provided in this application.

[0179] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0181] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0182] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for identifying live streamers, characterized in that, include: The acquired live content is subjected to video content acquisition and audio content acquisition respectively, resulting in video frame data and audio frame data; Mouth shape feature indicators are extracted from the video frame data to obtain the first pronunciation feature data of the lips; Speech activity detection and speech rate feature extraction are performed on the audio frame data to obtain the second pronunciation feature data of the lips and speech rate feature data; Based on the first pronunciation feature data and the second pronunciation feature data, the synchronization matching rate between the pronunciation mouth shape and the actual speech is determined; Based on the synchronization matching rate and the speech rate feature data, the live stream recognition result is obtained.

2. The live-streaming identification method according to claim 1, characterized in that, The step of extracting lip shape feature indicators from the video frame data to obtain the first pronunciation feature data of the lips includes: For each video frame, the mouth opening and closing index, the upper and lower lip area change index, and the mouth symmetry index are extracted respectively. For each video frame, the mouth opening and closing index data, upper and lower lip area change characteristic index data, and mouth symmetry characteristic index data of the video frame are fused according to a preset weight to obtain the index fusion data of the video frame. Based on the fusion data of the aforementioned indicators and the lip shape threshold, it is determined whether the lip shape in the video frame is a vocal lip shape.

3. The live-streaming identification method according to claim 1, characterized in that, The process of performing speech activity detection and speech rate feature extraction on the audio frame data to obtain the second pronunciation feature data of the lips and speech rate feature data includes: Speech activity detection is performed on each audio frame based on a speech activity detection algorithm to determine whether there is real speech in each audio frame. The speech rate feature data is determined based on the number of target audio frames contained in multiple time windows and the duration of the multiple time windows. The target audio frames are the audio frames in the multiple time windows in which the real speech exists.

4. The live-streaming identification method according to claim 3, characterized in that, The speech rate feature data is determined based on the number of target audio frames contained in multiple time windows and the duration of the multiple time windows, including: For each time window, the speech rate value is determined based on the number of target audio frames contained within the time window and the duration of the time window. The speech rate standard deviation is determined based on the speech rate value for each time window.

5. The live-streaming identification method according to claim 1, characterized in that, The step of determining the synchronization matching rate between the mouth shape and the actual speech based on the first pronunciation feature data and the second pronunciation feature data includes: The synchronization matching rate is determined based on the number of audio-visual synchronization frames and the number of audio frames identified as real speech. The audio-visual synchronization frames are video frames with mouth shapes that are vocalization mouth shapes and audio frames containing real speech at the same acquisition time.

6. The live-streaming identification method according to claim 4, characterized in that, The process of obtaining live-stream recognition results based on the synchronization matching rate and the speech rate feature data includes: If the synchronization matching rate is greater than the synchronization matching threshold, the speech rate is within the speed range, and the speech rate standard deviation is not less than the standard deviation threshold, then it is determined to be a live broadcast by a real person.

7. The live-streaming identification method according to claim 6, characterized in that, The process of obtaining the live-stream recognition result based on the synchronization matching rate and the speech rate feature data also includes: If the synchronization matching rate is not greater than the synchronization matching threshold, or the speech rate value is not within the speed value range, or the speech rate standard deviation is less than the standard deviation threshold, then corresponding risk management measures shall be taken according to the degree of deviation of the synchronization matching rate, or the degree of deviation of the speech rate value, or the degree of deviation of the speech rate standard deviation.

8. A live-streaming recognition device, characterized in that, include: The frame data acquisition module is used to acquire video content and audio content from the obtained live content, and obtain video frame data and audio frame data. The video frame processing module is used to extract mouth shape feature indicators from the video frame data to obtain the first pronunciation feature data of the lips; An audio frame processing module is used to perform speech activity detection and speech rate feature extraction on the audio frame data to obtain the second pronunciation feature data of the lips and speech rate feature data. The frame data matching module is used to determine the synchronization matching rate between the mouth shape and the actual speech based on the first pronunciation feature data and the second pronunciation feature data. as well as, The live-streaming recognition module is used to obtain the live-streaming recognition result based on the synchronization matching rate and the speech rate feature data.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the live-streaming recognition method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the live-streaming recognition method as described in any one of claims 1 to 7.

Images