AI-based speaker tracking method and system in live meeting

Abstract

This invention relates to the field of video conferencing technology, and more particularly to an AI-based speaker tracking method and system for live conferences. The method extracts acoustic and visual features from multiple audio and video streams, aligns multimodal features to determine a candidate speaker set. Combining spatial location, audio persistence, and historical state, a state transition rule is used to determine the current speaker and generate an identifier. Based on this, the main display video stream is switched, and the rules are dynamically adjusted according to the switching situation. This invention achieves accurate and smooth speaker tracking, improving the viewing experience and automation level of live conferences.

Description

Technical Field

[0001] This invention relates to the field of video conferencing technology, and in particular to an AI-based speaker tracking method and system for live conferences. Background Technology

[0002] In existing multi-person video conferencing or live streaming systems, automatic speaker tracking is a key requirement for improving the meeting experience and viewing quality. Common practices mainly rely on processing single audio or visual signals. One widely adopted approach is simple switching based on audio energy detection, which continuously monitors the energy level of each participant's audio channel and switches the video feed corresponding to the channel with the highest audio energy that consistently exceeds a preset threshold to the main display screen. Another approach combines preliminary person detection and facial orientation analysis, capturing frontal images of participants through a camera and supplementing this with simple motion detection to determine the likelihood of them speaking.

[0003] However, these conventional methods have significant limitations. In complex multi-person free discussion scenarios, background noise, multiple people speaking simultaneously, or brief interruptions frequently occur. Relying solely on audio energy peaks for judgment can easily lead to frequent and inappropriate switching of the screen between multiple participants, causing visual fatigue and distraction for viewers. At the same time, non-speech high-energy sounds such as a brief cough, throat clearing, or dropping objects can also be misinterpreted as the start of a speech, triggering incorrect video switching. On the other hand, relying solely on visual information such as facial orientation becomes significantly less reliable when participants are turned to the side, looking down at notes, or when multiple people are in the same frame, making it difficult to effectively distinguish who is speaking, and especially difficult to handle situations where the audio and video streams are not precisely aligned or there are transmission delays.

[0004] Furthermore, existing methods typically lack continuous modeling and smooth transition mechanisms for speaking states, often overreacting to instantaneous signal changes while ignoring the fact that speaking is usually a continuous process, leading to instability in the identified speakers. At the same time, conventional solutions rarely consider the speaker's historical state (such as having just finished speaking) to impose reasonable constraints, preventing the main screen from jumping to an irrelevant third party during brief intervals between two main speakers, which affects the continuity and professional feel of the meeting. How to accurately, stably, and smoothly track speakers automatically, while reducing erroneous switching and jitter, remains a problem that urgently needs improvement in current technology. Summary of the Invention

[0005] This invention provides an AI-based speaker tracking method and system for live conferences, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides an AI-based speaker tracking method for live conferences, comprising:

[0007] The system acquires multiple audio streams and multiple video streams in a live conference scenario, performs acoustic feature extraction on the multiple audio streams to obtain time-frequency domain feature representations of each audio stream, and performs visual feature extraction on the multiple video streams to obtain spatial location features and dynamic behavior features of people and objects in each video stream.

[0008] Based on the time-frequency domain feature representation and the dynamic behavior features, the correspondence between the audio stream and the video stream is determined, and an aligned multimodal feature sequence is obtained. Based on the aligned multimodal feature sequence, the person in the video stream corresponding to the audio stream that has vocalization behavior at the current moment is determined, and a candidate speaker set is obtained.

[0009] Based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking status information of each person in the candidate speaker set, the current speaker is determined through state transition constraint rules, and speaker identification information is generated.

[0010] Based on the speaker identification information, the main display video stream of the live conference is switched to the video stream corresponding to the current speaker object, and the state transition constraint rules are adjusted based on the playback duration and switching frequency of the switched video stream.

[0011] This involves acquiring multiple audio and video streams from a live conference scenario, performing acoustic feature extraction on the audio streams to obtain time-frequency domain feature representations for each audio stream, and performing visual feature extraction on the video streams to obtain spatial location and dynamic behavior features of objects in each video stream, including:

[0012] Each audio segment in the multi-channel audio stream undergoes time-frequency transformation processing to extract frequency domain and time domain representation features of the audio segment. The time-frequency domain feature representation is then generated by jointly processing the frequency domain and time domain representation features.

[0013] The multiple video streams are processed to detect human objects in the images, and the facial region coordinates of each human object are extracted to obtain the spatial region features of the human object;

[0014] For each character object, a deformation feature sequence of the lip region is extracted. After encoding the deformation feature sequence, a lip movement representation vector and lip movement periodicity features are obtained.

[0015] Based on the lip movement representation vector, and combined with the head posture change information and facial activation region distribution information of each person, a behavior association matrix is ​​determined;

[0016] The dynamic behavioral features are generated by modulating the lip movement representation vector based on the behavioral correlation matrix and the periodic features of lip movement.

[0017] Based on the time-frequency domain feature representation and the dynamic behavior features, the correspondence between the audio stream and the video stream is determined, resulting in an aligned multimodal feature sequence. Then, based on the aligned multimodal feature sequence, the person in the video stream corresponding to the audio stream exhibiting vocal behavior at the current moment is determined, resulting in a candidate speaker set, including:

[0018] Extract the temporal envelope curve of the energy change pattern in the time-frequency domain feature representation and the amplitude change curve of the lip movement feature in the dynamic behavior feature, calculate the temporal similarity between the temporal envelope curve and the amplitude change curve, and obtain a preliminary matching score between the audio stream and the video stream;

[0019] Based on the preliminary matching score, audio streams and video streams that meet the similarity constraints are paired, and the timestamp correspondence between the audio streams and video streams is obtained;

[0020] Based on the timestamp correspondence, the time-frequency domain feature representation, the spatial region features of the person / object, and the dynamic behavior features are aligned and fused to generate the aligned multimodal feature sequence;

[0021] Temporal analysis is performed on the aligned multimodal feature sequence to determine the target audio stream whose speech activity segment is identified as active in the time-frequency domain feature representation at the current time, and the target video stream mapped by the target audio stream in the timestamp correspondence.

[0022] Extract the person objects within the time period corresponding to the voice activity segment identifier from the target video stream, verify the consistency of the person objects' voices based on the spatial region features and dynamic behavior features, and obtain the speaker candidate set.

[0023] Extract the person objects within the time period corresponding to the speech activity segment identifier from the target video stream, verify the consistency of the person objects' vocalizations based on the spatial region features and dynamic behavior features, and obtain the speaker candidate set, including:

[0024] Extract all human objects within the time period corresponding to the voice activity segment identifier from the target video stream, and obtain the spatial region features and dynamic behavior features of each human object;

[0025] Based on the spatial region features of each person object, the spatial position distribution stability of each person object in the target video stream is calculated, and person objects whose spatial position distribution stability satisfies the visibility constraint are selected to obtain the spatial filtering result;

[0026] For each person in the spatial filtering results, extract the periodic features of lip movements from the dynamic behavior features, and calculate the temporal overlap between the periodic features of lip movements and the time range of the speech activity segment identifier;

[0027] For each person in the spatial filtering results, the facial activation pattern in the dynamic behavior features is extracted, and a multi-dimensional consistency evaluation index is determined by combining the periodic features of lip movement with the facial activation pattern.

[0028] Based on the multi-dimensional consistency evaluation index and the temporal overlap, the vocal consistency of each person is determined, and the person whose vocal consistency meets the consistency judgment condition is added to the speaker candidate set.

[0029] Based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking state information of each individual in the candidate speaker set, the current speaker is determined through state transition constraint rules, and speaker identification information is generated, including:

[0030] Based on the persistence characteristics represented by the time-frequency domain features, the duration of speech activity and the stability of speech energy of the audio stream corresponding to each speaker in the candidate speaker set are calculated to generate a persistence evaluation index;

[0031] Based on the historical speaking status information, the speaking frequency and speaking transition pattern of each individual in the speaker candidate set within the historical time window are extracted to determine the state transition probability distribution;

[0032] Based on the spatial location characteristics of each individual in the candidate speaker set, the spatial distance relationship and spatial location stability between each individual are calculated, and spatial constraints are generated.

[0033] Based on the continuous evaluation index, the state transition probability distribution, and the spatial constraints, the comprehensive score of each person in the speaker candidate set is determined using the state transition constraint rules;

[0034] The person with the highest overall score is selected as the current speaker, and the speaker identification information is generated based on the current speaker.

[0035] Based on the aforementioned continuous evaluation indicators, the aforementioned state transition probability distribution, and the aforementioned spatial constraints, the comprehensive score of each individual in the speaker candidate set is determined using the aforementioned state transition constraint rules, including:

[0036] Extract the duration of speech activity of each person in the audio stream corresponding to the duration of speech activity from the duration of speech activity, and calculate the duration of duration of speech activity based on the ratio of the duration of speech activity to the preset duration of duration of duration.

[0037] Extract the historical speech transition frequency of each character in the state transition probability distribution, and calculate the state transition penalty factor based on the historical speech transition frequency;

[0038] Extract the spatial positional stability of each character object under the spatial constraints, and calculate the spatial reliability coefficient based on the spatial positional stability.

[0039] The persistence score is obtained by fusing the speech energy stability of each individual in the persistence evaluation index with the persistence weight coefficient, and the historical state score is obtained by fusing the speaking frequency of each individual in the state transition probability distribution within the historical time window with the state transition penalty factor.

[0040] Based on the state transition constraint rules, the persistence score, the historical state score, and the spatial credibility coefficient are weighted and fused to obtain the comprehensive score of each person in the speaker candidate set.

[0041] Based on the speaker identification information, the main display video stream of the live conference is switched to the video stream corresponding to the current speaker object, and the state transition constraint rules are adjusted based on the playback duration and switching frequency of the switched video stream, including:

[0042] Based on the speaker identification information, the video stream corresponding to the current speaker object is identified, and the main display video stream of the live conference is switched from the currently playing video stream to the video stream corresponding to the current speaker object.

[0043] Record the start time of the video stream corresponding to the current speaker object as the main display video stream, and record the end time of the next video stream switch. Calculate the time difference between the start time and the end time to obtain the playback duration of the switched video stream.

[0044] The number of times the main display video stream is switched is counted within a preset statistical time window, and the switching frequency is calculated based on the number of switches and the duration of the preset statistical time window.

[0045] Based on the playback duration of the switched video stream and the switching frequency, rule adjustment parameters are determined. Based on the rule adjustment parameters, the persistence evaluation index and state transition penalty factor in the state transition constraint rules are adjusted to obtain the updated state transition constraint rules.

[0046] A second aspect of this invention provides an AI-based speaker tracking system for live conferences, comprising:

[0047] The feature extraction unit is used to acquire multiple audio streams and multiple video streams in a live conference scenario, perform acoustic feature extraction on the multiple audio streams to obtain time-frequency domain feature representations of each audio stream, and perform visual feature extraction on the multiple video streams to obtain spatial location features and dynamic behavior features of people and objects in each video stream.

[0048] The alignment analysis unit is used to determine the correspondence between the audio stream and the video stream based on the time-frequency domain feature representation and the dynamic behavior features, to obtain the aligned multimodal feature sequence, and to determine the person object in the video stream corresponding to the audio stream that has vocal behavior at the current moment based on the aligned multimodal feature sequence, so as to obtain a candidate speaker set.

[0049] The speaker identification unit is used to determine the current speaker object and generate speaker identification information based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking status information of each person object in the speaker candidate set, through state transition constraint rules.

[0050] The video control unit is used to control the main display video stream of the live conference to switch to the video stream corresponding to the current speaker object according to the speaker identification information, and to adjust the state transition constraint rules based on the playback duration and switching frequency of the switched video stream.

[0051] A third aspect of the present invention provides an electronic device, comprising:

[0052] processor;

[0053] Memory used to store processor-executable instructions;

[0054] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0055] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0056] This invention significantly improves the accuracy and smoothness of speaker identification and switching in live conferences. By simultaneously analyzing multiple audio and video streams and fusing acoustic and visual behavioral features, it can accurately establish audio-visual correspondences, thereby effectively filtering out candidates who are speaking. This method overcomes the misjudgments that may result from relying solely on audio energy or face detection, and can reliably identify potential speakers even in scenarios with multiple people active simultaneously or in the presence of environmental noise.

[0057] After determining the candidate speaker set, the solution further incorporates multi-dimensional information such as spatial location, speaking duration, and historical status for comprehensive decision-making. Spatial location features help to accurately locate the speaker in the image, duration features can filter out interfering sounds such as brief coughs or knocking, and historical status information ensures the consistency of speaking rights determination. By coordinating these factors through preset state transition constraint rules, the solution can intelligently determine the true dominant speaker and avoid unnecessary and frequent shifts in the focus of the speech among multiple speakers.

[0058] Ultimately, based on the judgment result, the main display screen is automatically switched to the current speaker, automating the broadcasting process. Simultaneously, the solution is adaptive, dynamically adjusting the thresholds or parameters of the state transition rules according to the actual playback duration and frequency of video stream switching. This mechanism allows it to learn and adapt to different meeting styles and rhythms, maintaining the rationality and naturalness of switching decisions in different modes such as free discussion or turn-based speaking. This reduces human intervention while ensuring a focused and immersive viewing experience for remote participants. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the AI-based speaker tracking method in a live conference according to an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram illustrating the process of generating speaker identification information according to an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0063] Figure 1 This is a flowchart illustrating an AI-based speaker tracking method for live video conferencing, as described in an embodiment of the present invention. Figure 1 As shown, AI-based speaker tracking methods in live conferences include:

[0064] The system acquires multiple audio streams and multiple video streams in a live conference scenario, performs acoustic feature extraction on the multiple audio streams to obtain time-frequency domain feature representations of each audio stream, and performs visual feature extraction on the multiple video streams to obtain spatial location features and dynamic behavior features of people and objects in each video stream.

[0065] Based on the time-frequency domain feature representation and the dynamic behavior features, the correspondence between the audio stream and the video stream is determined, and an aligned multimodal feature sequence is obtained. Based on the aligned multimodal feature sequence, the person in the video stream corresponding to the audio stream that has vocalization behavior at the current moment is determined, and a candidate speaker set is obtained.

[0066] Based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking status information of each person in the candidate speaker set, the current speaker is determined through state transition constraint rules, and speaker identification information is generated.

[0067] Based on the speaker identification information, the main display video stream of the live conference is switched to the video stream corresponding to the current speaker object, and the state transition constraint rules are adjusted based on the playback duration and switching frequency of the switched video stream.

[0068] This involves acquiring multiple audio and video streams from a live conference scenario, performing acoustic feature extraction on the audio streams to obtain time-frequency domain feature representations for each audio stream, and performing visual feature extraction on the video streams to obtain spatial location and dynamic behavior features of objects in each video stream, including:

[0069] Each audio segment in the multi-channel audio stream undergoes time-frequency transformation processing to extract frequency domain and time domain representation features of the audio segment. The time-frequency domain feature representation is then generated by jointly processing the frequency domain and time domain representation features.

[0070] The multiple video streams are processed to detect human objects in the images, and the facial region coordinates of each human object are extracted to obtain the spatial region features of the human object;

[0071] For each character object, a deformation feature sequence of the lip region is extracted. After encoding the deformation feature sequence, a lip movement representation vector and lip movement periodicity features are obtained.

[0072] Based on the lip movement representation vector, and combined with the head posture change information and facial activation region distribution information of each person, a behavior association matrix is ​​determined;

[0073] The dynamic behavioral features are generated by modulating the lip movement representation vector based on the behavioral correlation matrix and the periodic features of lip movement.

[0074] After the live conferencing system receives the audio and video data from the participants, it first performs time-frequency transformation processing on each audio segment in the multi-channel audio stream. Specifically, the sampling duration of each audio segment is set to 20 to 50 milliseconds, and the sampling rate is 16 kHz or 48 kHz. A short-time Fourier transform is performed on the audio segments to convert the time-domain signal to the frequency domain, resulting in a spectrogram representation. During the extraction of frequency domain characteristics, the spectrogram is divided into multiple frequency bands, each covering a specific frequency range, such as the low-frequency band (0-500 Hz), the mid-frequency band (500-2000 Hz), and the high-frequency band (2000-8000 Hz). The energy distribution and spectral envelope shape of each frequency band are calculated. Regarding the extraction of time-domain characteristics, the short-time energy, zero-crossing rate, and autocorrelation function characteristics of the audio segments are calculated. The short-time energy reflects the amplitude variation intensity of the audio signal, the zero-crossing rate reflects the frequency component distribution of the signal, and the autocorrelation function characteristics are used to capture the periodic patterns of the audio signal.

[0075] When jointly processing frequency domain and time domain representation features, a feature fusion mechanism is employed to combine the two types of features. Specifically, the frequency domain representation features are represented as feature vectors of dimension 128, and the time domain representation features are represented as feature vectors of dimension 64. These are then concatenated to form a joint feature vector of dimension 192. Subsequently, a fully connected layer is applied to the joint feature vector for nonlinear transformation, outputting a time-frequency domain feature representation of dimension 256. This time-frequency domain feature representation contains both the frequency component information of the audio and preserves the temporal variation characteristics, thus comprehensively characterizing the acoustic properties of the audio stream.

[0076] In video stream processing, human object detection is performed on each frame of multiple video streams. An object detection algorithm is used to locate the bounding boxes of human objects within the video frames, obtaining the pixel coordinate range of each human object. Specifically, the detection algorithm outputs the coordinates of the top-left and bottom-right corners of the bounding box for each human object, using these coordinates to determine the spatial position of the human object in the frame. The spatial region features are represented by the coordinates of the center point of the bounding box, the width and height of the bounding box, and the proportion of the bounding box area to the total frame area. After detecting human objects, facial region coordinates are further extracted. A facial keypoint detection algorithm is used to locate the positions of facial feature points, including the coordinate information of key areas such as the eyes, nose, and mouth. The extracted facial region coordinates are stored as a coordinate set containing 68 keypoints, each represented by a two-dimensional coordinate value.

[0077] When extracting the deformation feature sequence of the lip region for each person, a subset of keypoints related to the lips is first selected from facial keypoints. This subset typically contains 12 to 20 keypoints distributed around the upper lip edge, lower lip edge, and corners of the mouth. The positional changes of these lip keypoints are tracked in a continuous video frame sequence, and the displacement vectors of the keypoints between adjacent frames are calculated. The deformation feature sequence consists of a series of displacement vectors, each reflecting the trend of lip shape change over time. To quantify the degree of lip opening and closing, the vertical distance between the center points of the upper and lower lip is calculated; this distance serves as an important deformation indicator in the deformation feature sequence. Simultaneously, the horizontal distance between the corners of the mouth is calculated to characterize the lateral expansion state of the lips.

[0078] When encoding the deformation feature sequence, a temporal coding network is used to process it. The temporal coding network receives deformation features from several consecutive frames as input and extracts the temporal pattern of the deformation features through a combination of convolutional layers and recurrent neural network layers. The encoding process outputs a lip movement representation vector, with a dimension of 128, which compresses and represents the overall characteristics of lip movement. Simultaneously, the periodic features of lip movement are extracted during the encoding process. Specifically, frequency domain analysis is performed on the deformation feature sequence to calculate the power spectral density and identify the peak frequency components within the power spectral density. The periodic features of lip movement are represented by the dominant frequency value and the corresponding period length, reflecting the rhythmic characteristics of lip movement during speech.

[0079] When determining the behavior association matrix, it is necessary to comprehensively consider the lip movement representation vector, head posture change information, and facial activation region distribution information. Head posture change information is obtained by calculating the rotation angles of the subject's head in three-dimensional space, including pitch, yaw, and roll angles. Head posture extraction relies on the three-dimensional reconstruction of facial key points, estimating the head's spatial posture relative to the camera using a perspective projection model. Facial activation region distribution information is obtained by analyzing the muscle movement intensity in different facial regions. The face is divided into eyebrow, eye, nose, lip, and cheek regions, and the movement amplitude of key points in each region is calculated to form an activation intensity distribution map.

[0080] The behavioral association matrix is ​​constructed by concatenating the vectors representing lip movements, the encoded vectors of head posture changes, and the encoded vectors of facial activation region distribution to form a comprehensive behavioral feature vector. This comprehensive feature vector is then subjected to an outer product operation or an attention mechanism is used to calculate the association weights between features, generating a matrix structure with several rows and columns. Each element in the behavioral association matrix reflects the strength of the collaborative relationship between different behavioral features; a larger value indicates a stronger association between the corresponding features.

[0081] When modulating the lip movement representation vector based on the behavioral correlation matrix and the periodicity of lip movements, the behavioral correlation matrix is ​​first converted into weight coefficients, and the weight vector is obtained through row summation or column summation of the matrix. The weight vector is then multiplied element-wise with the lip movement representation vector to achieve weighted modulation of the features. The periodicity of lip movements is used as the modulation parameter, and the modulated features are normalized to a time scale based on the period length. Specifically, when a short lip movement period is detected, the weight of high-frequency components is increased; when a long lip movement period is detected, the weight of low-frequency components is increased. The output after the modulation operation is the dynamic behavioral feature. This feature vector maintains a dimension of 128 and integrates the spatial morphological information of lip movements, temporal evolution patterns, and their synergistic relationship with head posture and facial expressions, accurately representing the comprehensive dynamic behavioral pattern of a person during speech.

[0082] Through the above processing steps, rich feature representations can be extracted from multiple audio and video streams. Time-frequency domain feature representations comprehensively characterize the acoustic properties of the audio streams, spatial location features pinpoint the position of the speaker within the frame, and dynamic behavior features capture the speaker's visual behavioral patterns during speech. These features provide a solid data foundation for subsequent audio-video alignment and speaker recognition, enabling accurate determination of the current speaking status and achieving precise speaker tracking.

[0083] Based on the time-frequency domain feature representation and the dynamic behavior features, the correspondence between the audio stream and the video stream is determined, resulting in an aligned multimodal feature sequence. Then, based on the aligned multimodal feature sequence, the person in the video stream corresponding to the audio stream exhibiting vocal behavior at the current moment is determined, resulting in a candidate speaker set, including:

[0084] Extract the temporal envelope curve of the energy change pattern in the time-frequency domain feature representation and the amplitude change curve of the lip movement feature in the dynamic behavior feature, calculate the temporal similarity between the temporal envelope curve and the amplitude change curve, and obtain a preliminary matching score between the audio stream and the video stream;

[0085] Based on the preliminary matching score, audio streams and video streams that meet the similarity constraints are paired, and the timestamp correspondence between the audio streams and video streams is obtained;

[0086] Based on the timestamp correspondence, the time-frequency domain feature representation, the spatial region features of the person / object, and the dynamic behavior features are aligned and fused to generate the aligned multimodal feature sequence;

[0087] Temporal analysis is performed on the aligned multimodal feature sequence to determine the target audio stream whose speech activity segment is identified as active in the time-frequency domain feature representation at the current time, and the target video stream mapped by the target audio stream in the timestamp correspondence.

[0088] Extract the person objects within the time period corresponding to the voice activity segment identifier from the target video stream, verify the consistency of the person objects' voices based on the spatial region features and dynamic behavior features, and obtain the speaker candidate set.

[0089] In the actual operation of live streaming conferences, establishing a multimodal correspondence between the time-frequency domain feature representation of the audio stream and the dynamic behavioral features of the video stream requires precise analysis of the synchronization between sound and visual signals in the time dimension. For the energy change patterns in the time-frequency domain feature representation, a short-time energy statistical method is used to calculate the frame-level energy of the audio signal. Each frame length is set to 20 to 30 milliseconds, and the frame shift is set to 10 milliseconds. The energy value of each frame of audio data is calculated, and the energy values ​​of consecutive frames are arranged in chronological order to form a time envelope curve. This time envelope curve reflects the energy fluctuations of the audio signal on the time axis. When a participant speaks, the energy value of the corresponding time period will rise significantly, forming an energy peak region. When extracting lip movement features from the dynamic behavioral features, a facial keypoint detection algorithm is used to locate the lip region of the face, obtain the coordinates of the key points of the upper and lower lips, and calculate the vertical distance between the upper and lower lips as a quantitative indicator of the degree of lip opening. The degree of lip opening in consecutive video frames is arranged in chronological order to form an amplitude variation curve. This amplitude variation curve describes the dynamic changes in a person's lip movements. When a person speaks, their lips will produce continuous opening and closing movements, and the curve exhibits periodic fluctuation characteristics.

[0090] When calculating the temporal similarity between the time envelope curve and the amplitude variation curve, a dynamic time warping algorithm is used to perform matching analysis on the two curves. This algorithm can handle the nonlinear time alignment problem between time series and finds the optimal matching path by constructing a cumulative distance matrix. Specifically, the time envelope curve is represented as a curve of length [missing information]. sequence The amplitude variation curve is represented as a curve of length [missing information]. sequence Calculate the Euclidean distance between any two elements in the sequence, and construct a sequence of size . The distance matrix is ​​calculated. The minimum cumulative distance from the starting point to the ending point is calculated using dynamic programming, and the normalized value of this distance is used as a measure of temporal similarity. A high temporal similarity value indicates that the changes in audio energy and lip movement amplitude are well synchronized in time, and a preliminary matching score between the audio stream and the video stream is calculated based on this. This preliminary matching score comprehensively considers the similarity of the curve shape and the degree of optimization of temporal alignment.

[0091] When performing pairing analysis on audio and video streams based on preliminary matching scores, similarity constraints are set as the criteria for determining valid pairings. These constraints include a minimum temporal similarity threshold and a stability requirement for the matching scores. When the preliminary matching score between an audio stream and a video stream exceeds the set threshold and remains stable over multiple consecutive time windows, the audio stream and video stream are considered to be validly paired. In actual processing, a sliding window mechanism is used to segment the time series for analysis, with window lengths set to 1 to 3 seconds. Matching scores are calculated for the audio and video data within each window, and the stability of the pairing relationship is confirmed by statistically analyzing the scores over multiple consecutive windows. In cases where multiple video streams produce high matching scores with the same audio stream, the video stream with the highest score and the smallest fluctuation is selected as the pairing target for that audio stream. This pairing process establishes a timestamp correspondence between the audio and video streams, which records the synchronization relationship between each audio stream and its paired video stream on the timeline.

[0092] When aligning and fusing multimodal features based on the established timestamp correspondence, the timeline of the audio stream is used as the reference system. Spatial region features and dynamic behavior features in the video stream are mapped to corresponding positions on the audio timeline according to timestamps. For each frame of audio features in the time-frequency domain feature representation, the corresponding video frame is found according to the timestamp correspondence. Spatial region features of the person are extracted from the video frame, including the bounding box coordinates, center point coordinates, and proportion of the person in the frame. Simultaneously, dynamic behavior features of the person in the video frame are extracted, including lip movement amplitude, head posture angle, facial expression parameters, and other feature vectors describing the person's action state. The time-frequency domain feature representation of the audio, the spatial region features of the video, and the dynamic behavior features are concatenated according to a unified timestamp to form a fused feature vector containing multimodal information. This fused feature vector forms a continuous sequence in the time dimension, which is the aligned multimodal feature sequence. Each element in this sequence corresponds to the joint audio-video feature representation at a certain moment, providing a unified feature foundation for subsequent speaker recognition.

[0093] When performing time-series analysis on the aligned multimodal feature sequences, speech activity detection is first performed on the time-frequency domain feature representation to determine whether the audio signal at each moment contains valid speech components. Speech activity detection is comprehensively judged by analyzing multiple dimensions such as audio energy, zero-crossing rate, and spectral characteristics. When the audio energy exceeds a set multiple of the background noise level and the spectral distribution conforms to the typical characteristics of a speech signal, that moment is marked as a speech activity segment, and the corresponding speech activity segment identifier is set to active state. By identifying the speech activity segment identifiers of consecutive frames, it is possible to determine which audio streams are in the phonation state at the current moment, and these active audio streams are marked as target audio streams. For each target audio stream, its paired video stream is found according to the previously established timestamp correspondence; this video stream is the target video stream. After determining the target video stream, it is necessary to further locate the actual person speaking in the video stream.

[0094] The system extracts consecutive video frames within the time segment corresponding to the speech activity segment identifier from the target video stream. For each frame, it performs person detection and tracking to obtain the spatial region features of all person objects in the frame. These features include the bounding box position, size, and spatial distribution information relative to the frame for each person. To maintain consistency in the person's identity across consecutive frames, a target tracking algorithm is used to address positional changes, assigning a unique identifier to each person. For each person object in the frame, dynamic behavioral features within the corresponding time segment are extracted, with a focus on lip movement features. By calculating the degree of lip opening, movement speed, and periodicity, it is determined whether the person is making a vocalization. Simultaneously, head posture information is incorporated to eliminate cases where lip obstruction or unclear observation is caused by camera angle.

[0095] To verify the consistency of speech patterns among individuals, it is necessary to cross-validate visual lip movement features with auditory audio features. Specifically, for a given individual, the start and end times of their lip movement features should be temporally consistent with the speech activity segment of the corresponding audio stream. The time difference between the start time of the lip movement and the start time of the speech activity segment is calculated. When the time difference is less than a preset synchronization tolerance threshold, the individual's visual behavior and auditory signal are considered to have temporal consistency. Simultaneously, the changing trends of lip movement amplitude and audio energy envelope are analyzed; both should exhibit similar dynamic patterns. For individuals that simultaneously satisfy temporal synchronization and dynamic similarity, their comprehensive confidence score is calculated. This score comprehensively considers temporal alignment, feature similarity, and spatial stability. All individuals that pass the speech consistency verification are sorted according to their confidence scores to form a candidate speaker set. This candidate set includes individuals with a higher probability of currently speaking and their corresponding confidence assessment results, providing a filtered candidate range for subsequent speaker determination. Through the above-mentioned multi-level feature alignment, temporal analysis and consistency verification process, deep fusion of audio and video multimodal information and accurate identification of speaker candidates are achieved.

[0096] Extract the person objects within the time period corresponding to the speech activity segment identifier from the target video stream, verify the consistency of the person objects' vocalizations based on the spatial region features and dynamic behavior features, and obtain the speaker candidate set, including:

[0097] Extract all human objects within the time period corresponding to the voice activity segment identifier from the target video stream, and obtain the spatial region features and dynamic behavior features of each human object;

[0098] Based on the spatial region features of each person object, the spatial position distribution stability of each person object in the target video stream is calculated, and person objects whose spatial position distribution stability satisfies the visibility constraint are selected to obtain the spatial filtering result;

[0099] For each person in the spatial filtering results, extract the periodic features of lip movements from the dynamic behavior features, and calculate the temporal overlap between the periodic features of lip movements and the time range of the speech activity segment identifier;

[0100] For each person in the spatial filtering results, the facial activation pattern in the dynamic behavior features is extracted, and a multi-dimensional consistency evaluation index is determined by combining the periodic features of lip movement with the facial activation pattern.

[0101] Based on the multi-dimensional consistency evaluation index and the temporal overlap, the vocal consistency of each person is determined, and the person whose vocal consistency meets the consistency judgment condition is added to the speaker candidate set.

[0102] In determining the candidate speaker set, it is necessary to accurately identify the actual speakers from the target video stream. First, all human objects in the target video stream are extracted from the time period corresponding to the speech activity segment identifier. The target video stream refers to the video stream corresponding to the audio stream where speech is currently occurring, after audio-video alignment. The speech activity segment identifier records the start and end times of the detected speech period. Within this time period, a deep learning-based object detection algorithm is used to detect human objects in each frame of the video stream, obtaining the bounding box information of all human figures in the frame. For each detected human object, its spatial coordinates, bounding box size, and relative position in the frame are recorded. Simultaneously, dynamic behavioral features of each human object are extracted, including changes in facial keypoint positions, lip region morphology, and head posture.

[0103] To extract spatial region features, it is necessary to analyze the spatial position distribution of each person object in consecutive frames of the target video stream. Spatial region features include the center coordinates of the object's bounding box, the width and height of the bounding box, the pixel area occupied by the object in the frame, and the object's offset relative to the frame center. To calculate the stability of the spatial position distribution, for consecutive video frames within a time window, the position coordinate sequence of each person object in each frame is extracted, and the variance and standard deviation of this position sequence are calculated. The stability of the spatial position distribution is evaluated by analyzing the fluctuation range of the position coordinates. When the change in the position coordinates of a person object in consecutive frames is small, it indicates that its spatial position distribution is stable. Specifically, for the number of video frames included in the time window, the range of change of the x-coordinate and y-coordinate of the object's center point is statistically analyzed. When the range of change of both the x-coordinate and y-coordinate is less than a preset threshold, the spatial position distribution of the person object is considered stable.

[0104] Visibility constraints are used to determine whether a person or object maintains sufficient visibility within the image. These constraints include several evaluation dimensions. The first dimension is the object's completeness within the image, determined by checking if its bounding box is clipped by the image edges. A person's bounding box is considered complete when it is entirely within the image and its distance from the image edge is greater than a set edge distance threshold. The second dimension is the object's occlusion level, assessed by analyzing the pixel completeness of the face region. An occlusion constraint is considered satisfied when the proportion of visible pixels in the face region is higher than an occlusion assessment threshold. The third dimension is the object's size proportion, i.e., the ratio of the object's bounding box to the total pixel area of ​​the image. A size constraint is satisfied when this ratio is within a reasonable range—neither too small (resulting in unclear details) nor too large (resulting in incomplete facial information). Only person or object objects that simultaneously meet all three dimensions are included in the spatial filtering results.

[0105] After obtaining the spatial filtering results, dynamic behavioral feature analysis was performed on each individual subject. The periodic features of lip movements reflect the regular changes in lip shape during speech. To extract these periodic features, key lip points were first located in each frame, including the coordinates of the upper lip apex, lower lip base, and corners of the mouth. The time series of lip opening and closing amplitude was calculated by analyzing the positional changes of these key lip points across consecutive frames. This time series reflects the dynamic changes in lip opening and closing. Frequency domain analysis was then performed on the time series of lip opening and closing amplitude to extract its dominant frequency components and energy distribution characteristics. During normal speech, lip movements exhibit obvious periodic characteristics, and their frequency range typically corresponds to the syllable frequencies of the speech signal.

[0106] Temporal overlap is used to measure the degree of matching between active periods of lip movement and the phonation periods recorded by speech activity segment identifiers. To calculate temporal overlap, the time periods where lip movement amplitude exceeds a threshold are first identified; these periods represent moments of significant lip movement. Then, this time period is compared with the time range of the speech activity segment identifiers. Temporal overlap is obtained by calculating the ratio of the intersection length to the union length of the two time periods. When the active periods of lip movement highly overlap with the phonation periods, the temporal overlap is close to 1, indicating that the lip movements of the subject are highly synchronized with the audio signal in time.

[0107] Facial activation patterns describe the comprehensive features of facial expressions and muscle movements during speech. When extracting facial activation patterns, in addition to focusing on the lip area, it is also necessary to analyze the dynamic changes in other facial areas, including cheek muscle movement, jaw opening and closing amplitude, changes in the eyebrow and forehead areas, and changes in the state of the eye area. A facial motion coding system encodes the movements of different facial areas into feature vectors. These feature vectors reflect the coordinated movement patterns of various facial areas during speech production. During normal speech, lip movements are accompanied by coordinated changes in the cheeks, jaw, and other areas, forming specific facial activation patterns.

[0108] The multidimensional consistency assessment index comprehensively considers the synergy between the periodic characteristics of lip movements and facial activation patterns. When establishing the multidimensional consistency assessment index, the periodic intensity of lip movements is first quantified by analyzing the autocorrelation function of the time series of lip opening and closing amplitudes to calculate its periodic score. Then, facial activation patterns are scored, and a facial synergy score is calculated based on the degree of synergy between the movement amplitudes of areas such as the cheeks and jaw and lip movements. The multidimensional consistency assessment index is obtained by weighted summation of the periodic score and the facial synergy score. The weighting is determined based on the importance of different regions in vocalization, with the lip region typically having a higher weight than other facial regions.

[0109] When determining the vocal consistency of individuals, a comprehensive consideration is given to multi-dimensional consistency evaluation indicators and temporal overlap. The calculation of vocal consistency combines multiple verifications of spatial, dynamic, and temporal features. For each individual, if its multi-dimensional consistency evaluation indicators exceed the set consistency scoring threshold and its temporal overlap exceeds the set temporal synchronization threshold, the individual is considered to have high vocal consistency. The consistency judgment criteria are set according to the requirements of the actual application scenario. In scenarios requiring high precision, the thresholds are set more strictly to reduce false positives; in scenarios with high real-time requirements, the thresholds are set more leniently to ensure response speed. Through the above multi-level screening and verification process, all individuals that meet the consistency judgment criteria are added to the speaker candidate set, providing a reliable set of candidates for subsequent speaker determination.

[0110] Figure 2 This is a schematic diagram of the process for generating speaker identification information according to an embodiment of the present invention, such as... Figure 2 As shown, based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking state information of each individual in the candidate speaker set, the current speaker is determined through state transition constraint rules, and speaker identification information is generated, including:

[0111] Based on the persistence characteristics represented by the time-frequency domain features, the duration of speech activity and the stability of speech energy of the audio stream corresponding to each speaker in the candidate speaker set are calculated to generate a persistence evaluation index;

[0112] Based on the historical speaking status information, the speaking frequency and speaking transition pattern of each individual in the speaker candidate set within the historical time window are extracted to determine the state transition probability distribution;

[0113] Based on the spatial location characteristics of each individual in the candidate speaker set, the spatial distance relationship and spatial location stability between each individual are calculated, and spatial constraints are generated.

[0114] Based on the continuous evaluation index, the state transition probability distribution, and the spatial constraints, the comprehensive score of each person in the speaker candidate set is determined using the state transition constraint rules;

[0115] The person with the highest overall score is selected as the current speaker, and the speaker identification information is generated based on the current speaker.

[0116] When evaluating candidates, the evaluation first focuses on the persistence characteristics of the time-frequency domain features. For each candidate's audio stream, a sliding analysis window is set in the time domain. The window length depends on the duration of typical speech segments in the actual application scenario, typically set to 300 to 1000 milliseconds in a conference setting. Within this window, short-time energy is calculated on the audio signal, and the energy distribution curve is obtained by accumulating the square of the amplitude at each sampling point. The calculation of the duration of speech activity requires setting an energy threshold. When the short-time energy of multiple consecutive analysis windows is higher than this threshold, it is considered a valid speech activity state. The time span from the start of the first window exceeding the threshold to the end of the last window exceeding the threshold is recorded. To improve robustness, a hysteresis comparison mechanism is introduced in the threshold determination process. Two threshold boundaries are set: when the energy rises from below the lower threshold to above the upper threshold, it is determined as the start of speech; when the energy falls from above the upper threshold to below the lower threshold, it is determined as the end of speech. This mechanism effectively avoids misjudgments caused by noise fluctuations.

[0117] The assessment of speech energy stability is achieved by calculating the coefficient of variation (COP) of energy values ​​within an analytical window sequence. Specifically, for a candidate subject, the COP is calculated as the ratio of the mean energy value to the standard deviation of energy across all valid speech windows within a specific time period. A smaller COP indicates a relatively stable energy distribution in the audio stream, suggesting that the corresponding speech is more likely to be continuous rather than brief background noise or non-speech behaviors such as coughing. At the frequency domain level, the audio signal is converted to the frequency domain using a Fast Fourier Transform (FFT) to extract the frequency bands where the main energy is concentrated. The fundamental frequency of human speech typically falls between 85 Hz and 255 Hz. By analyzing the energy proportion within this band and the smoothness of the spectral envelope, it is possible to further determine whether the audio stream represents normal speech. A weighted fusion of the duration in the time domain with energy stability and the spectral characteristics in the frequency domain is used to form a sustained evaluation index. A higher index indicates a more stable and reliable speech state for the corresponding candidate subject.

[0118] To utilize historical speaking status information, a time window mechanism is established to trace the historical speaking patterns of each candidate. The length of the time window is dynamically adjusted according to the pace of the meeting; it is shortened during intense discussions to quickly respond to speaker switching, and extended during calmer speaking phases to maintain tracking stability. Within the historical time window, the number of times each candidate becomes the current speaker is counted as the speaking frequency feature. Candidates with high speaking frequency are more likely to continue speaking in subsequent moments, which aligns with the actual pattern in meetings where speakers tend to express their views fully. The extraction of speaking transition patterns focuses on the changing patterns of speaker identities between adjacent moments, recording the frequency of switching from one candidate to another by constructing a state transition matrix. Each element of this matrix represents the conditional probability of switching from candidate i to candidate j, obtained through statistical normalization of historical data. In practical applications, to prevent insufficient generalization ability due to excessively sparse matrices, a smoothing technique is introduced to assign a small non-zero probability to unobserved transition patterns.

[0119] The determination of the state transition probability distribution comprehensively considers the historical speaking frequency of the current candidate and the probability values ​​in the transition matrix. For the current set of candidate speakers, if the speaker from the previous time step is still in the candidate set, the probability of that speaker maintaining the speaking state is determined based on the corresponding self-transition probability in the transition matrix. For other candidate speakers, the probability of becoming the new speaker is determined based on the transition probability from the previous speaker to that speaker. In the initial stage where there is no clear information about the previous speaker, the normalized value of the historical speaking frequency of each candidate is used as the prior probability distribution. This probability distribution reflects the temporal continuity of speaking states in the meeting and avoids frequent incorrect speaker switching.

[0120] The analysis of spatial location features is based on the detection results of human objects in the video stream. For each detected human object, its bounding box coordinates in the video frame are extracted, typically using the center point or bottom center point of the bounding box as the representative spatial location. The Euclidean distance between each human object in the candidate speaker set is calculated to construct a spatial distance matrix. In fixed scenes such as conference rooms, speakers are usually located in specific spatial areas, such as in front of the conference table or at the podium. By statistically analyzing the spatial location distribution of historical speakers, high-probability speaking areas can be identified. When a candidate object is located within this area, its spatial constraint score increases accordingly. Spatial location stability is evaluated by calculating the change in position of the candidate object between consecutive frames. Drastic position changes correspond to the object being moved rather than speaking steadily, thus reducing the likelihood of it becoming the current speaker. For multi-camera shooting scenes, the spatial coordinates from different perspectives need to be transformed to a unified world coordinate system, and coordinate alignment is achieved through camera calibration parameters and homography transformation.

[0121] After obtaining the persistence evaluation index, state transition probability distribution, and spatial constraints, a comprehensive score is calculated using state transition constraint rules. These rules are designed as a multi-dimensional weighted fusion model, assigning weight coefficients to the features across the three dimensions. The weight of the persistence evaluation index reflects the fundamental role of speech activity features in speaker identification and is typically set to a higher value to ensure that individuals with clear vocalization behavior are prioritized. The weight of the state transition probability reflects the importance of temporal coherence, preventing jumps caused by transient noise or false detections. The weight of the spatial constraints is adjusted according to the scenario characteristics; it is higher in a meeting room with fixed seating and appropriately lower in a freely moving scenario. Before fusion, the feature values ​​of each dimension need to be normalized to map values ​​with different dimensions to a unified numerical range. Common methods include maximum-minimum normalization or Z-score standardization.

[0122] The comprehensive score is calculated by weighting and summing the normalized persistence evaluation index, state transition probability, and spatial constraint score. For each candidate speaker, the comprehensive score reflects the overall confidence level of that speaker at the current moment. To improve decision reliability, a threshold mechanism is set based on the comprehensive score. The current speaker is only determined when the highest score exceeds the preset threshold; otherwise, the speaker status from the previous moment is maintained or the speaker is marked as having no clear speaker. The threshold setting needs to strike a balance between response speed and stability. Too low a threshold leads to frequent false switches, while too high a threshold results in delayed responses to new speakers. In actual deployment, the threshold can be dynamically adjusted according to the meeting type: a higher threshold is used for formal reporting meetings to maintain stability, while a lower threshold is used for discussion and interactive meetings to improve response sensitivity.

[0123] After identifying the speaker with the highest overall score as the current speaker, corresponding speaker identification information is generated. This identification information includes the speaker's unique identifier, position coordinates in the video stream, current timestamp, and confidence score. The identifier can be a combination of the video stream number and the speaker detection box number to ensure uniqueness in multi-camera, multi-person scenarios. The position coordinates are used for subsequent video cropping and image adjustment. The timestamp records the exact moment the speaker was identified, used for audio-visual synchronization and historical status updates. The confidence score is the numerical value of the overall score, which can be used for subsequent visualization or decision optimization. The generated speaker identification information is passed to the live streaming system's video switching module, triggering an update operation in the main display video stream. Simultaneously, this information is stored in the historical speaking status database as input data for calculating the state transition probability at subsequent moments, forming a closed-loop feedback mechanism.

[0124] Based on the aforementioned continuous evaluation indicators, the aforementioned state transition probability distribution, and the aforementioned spatial constraints, the comprehensive score of each individual in the speaker candidate set is determined using the aforementioned state transition constraint rules, including:

[0125] Extract the duration of speech activity of each person in the audio stream corresponding to the duration of speech activity from the duration of speech activity, and calculate the duration of duration of speech activity based on the ratio of the duration of speech activity to the preset duration of duration of duration.

[0126] Extract the historical speech transition frequency of each character in the state transition probability distribution, and calculate the state transition penalty factor based on the historical speech transition frequency;

[0127] Extract the spatial positional stability of each character object under the spatial constraints, and calculate the spatial reliability coefficient based on the spatial positional stability.

[0128] The persistence score is obtained by fusing the speech energy stability of each individual in the persistence evaluation index with the persistence weight coefficient, and the historical state score is obtained by fusing the speaking frequency of each individual in the state transition probability distribution within the historical time window with the state transition penalty factor.

[0129] Based on the state transition constraint rules, the persistence score, the historical state score, and the spatial credibility coefficient are weighted and fused to obtain the comprehensive score of each person in the speaker candidate set.

[0130] After obtaining the continuous evaluation indicators, state transition probability distribution, and spatial constraints, a comprehensive scoring mechanism is needed to integrate these multi-dimensional features into a unified decision-making basis. This process uses state transition constraint rules to quantitatively evaluate each candidate speaker, thereby providing reliable support for the final speaker selection.

[0131] For the processing of the persistence evaluation index, the duration of speech activity in the audio stream corresponding to each person is first extracted. This duration reflects the time span during which a specific person maintains a continuous vocal state, obtained by analyzing the continuous time periods in the audio signal where the energy envelope curve is above a preset threshold. In actual calculations, an energy detection window is set for the audio stream of each person, with a window length of 30 to 50 milliseconds. This parameter range is determined based on the short-term stationary characteristics of speech signals. According to speech signal processing theory, speech signals can be considered stationary within 10-30 milliseconds. Considering the balance between computational efficiency and detection accuracy, experiments were conducted on 100 sets of audio samples from different conference scenarios to statistically analyze the speech activity detection accuracy under different window lengths. The results show that the accuracy reaches 92% with a window length of 30 milliseconds and 89% with a window length of 50 milliseconds. However, the computational cost increases significantly when the window length is less than 30 milliseconds, and the response delay to short speech segments is obvious when the window length is greater than 50 milliseconds. Therefore, a window length of 40 milliseconds is preferred. In practical applications, the window length can be dynamically adjusted according to the audio sampling rate: when the sampling rate is 16kHz, the window length is set to 30 milliseconds (corresponding to 480 sampling points); when the sampling rate is 48kHz, the window length is set to 50 milliseconds (corresponding to 2400 sampling points) to ensure that each window contains enough sampling points for energy calculation. When the energy values ​​of multiple consecutive windows exceed the dynamic energy threshold, it is determined to be a valid speech activity. The time periods of these valid speech activities are accumulated to obtain the duration of the speech activity of the person in the current time window. The preset duration benchmark is dynamically adjusted according to the meeting type. It is set to 3 to 5 seconds for formal meeting scenarios and 1.5 to 2.5 seconds for discussion-style meetings. The setting of this benchmark duration is based on statistical analysis of 500 meetings of different types, collecting duration distribution data of valid speaking segments in various meeting scenarios such as formal speeches, group discussions, and Q&A interactions. Statistical results show that in formal presentations, the median duration of a single effective speaking segment is 4.2 seconds, with 75% of segments exceeding 3 seconds. Therefore, the baseline duration for formal meetings is set at 4 seconds, allowing for adjustments between 3 and 5 seconds to accommodate different speaking styles. In discussion-style meetings, the median duration of a single effective speaking segment is 2.1 seconds, with 75% of segments exceeding 1.5 seconds. Therefore, the baseline duration for discussion-style meetings is set at 2 seconds, allowing for adjustments between 1.5 and 2.5 seconds. In practical applications, the system monitors the average speaking duration over the past 5 minutes in real time. When the average duration consistently exceeds 4 seconds, it is classified as a formal meeting and a longer baseline duration is adopted; when the average duration is below 3 seconds, it is classified as a discussion-style meeting and a shorter baseline duration is adopted; when the average duration is between 3 and 4 seconds, a linear interpolation method is used to calculate the current baseline duration.The ratio of the duration of the speech activity to a preset duration benchmark is calculated. When this ratio is close to or exceeds 1, it indicates that the speaker has stable speaking behavior. Based on this ratio, a duration weighting coefficient is calculated, specifically through a sigmoid function mapping, which maps the ratio to the range of 0 to 1, so that audio streams with longer durations receive higher weighting coefficients.

[0132] For processing the state transition probability distribution, the historical speaking frequency of each participant was extracted as a key feature. The historical speaking frequency is calculated by counting the number of times a specific participant transitioned from a non-speaking state to a speaking state within a past time window; this indicator reflects the frequency of switching of speaking rights. The historical statistical time window was set to 30 to 60 seconds, determined based on the characteristics of human short-term memory and the analysis of meeting rhythm. Cognitive psychology research shows that human memory of recent events maintains high accuracy and completeness within 30 to 60 seconds; events exceeding this time range should have a lower weight in influencing current decisions. Analysis of speaking patterns in 300 meetings shows that in normal discussion scenarios, the switching pattern of speaking rights within 30 to 60 seconds effectively reflects the dynamic characteristics of the meeting. The specific value rules are as follows: For fast-paced discussion scenarios (average speaking time less than 10 seconds per session), the historical statistics window is set to 30 seconds to quickly respond to frequent speaking switches; for medium-paced scenarios (average speaking time 10 to 20 seconds per session), the historical statistics window is set to 45 seconds; for slow-paced formal scenarios (average speaking time greater than 20 seconds per session), the historical statistics window is set to 60 seconds to fully collect historical data on long speaking sessions. The average duration of recent speaking sessions is reassessed every 10 seconds, and the length of the historical statistics window is dynamically adjusted accordingly. Within this window, the number of times each person becomes the main speaker and the duration of that time are recorded. For frequently switching persons, their historical speaking transition frequency is high. This often corresponds to fast-paced Q&A or intense discussion scenarios, but it can also lead to excessively frequent screen switching, affecting the viewing experience. A state transition penalty factor is calculated based on the historical speaking transition frequency. This factor imposes a penalty on frequent switching behavior, reducing the probability of being selected as the main speaker at the current moment. The calculation of the state transition penalty factor uses an exponential decay function. When the historical speaking transition frequency exceeds a preset threshold, the penalty factor decreases exponentially, thereby suppressing excessive screen switching. The preset threshold is dynamically adjusted according to the pace of the meeting, set to 3 to 5 switches per minute for normal discussion scenarios. In practical applications, the actual switching frequency in the last 2 minutes is first counted. If the actual switching frequency is less than 3 times per minute, the state transition penalty factor is reduced to make it easier to accept new speaker candidates; if the actual switching frequency is more than 5 times per minute, the state transition penalty factor is increased to increase the difficulty of switching and stabilize the screen.

[0133] The handling of spatial constraints involves evaluating the spatial stability of each person / object. Spatial stability is obtained by analyzing the magnitude of positional changes of the person / object across consecutive video frames, specifically by calculating the variance or standard deviation of the center point coordinates of the person / object's bounding box over time. When a person / object remains relatively stationary, its positional coordinates change little, indicating high spatial stability and a stable speaking posture. Conversely, frequent changes in positional coordinates correspond to movement, turning, or other non-speaking actions, indicating low spatial stability. To quantify spatial stability, a movement speed threshold is set; a stable state is defined when the displacement between consecutive frames is below this threshold. A spatial reliability coefficient is calculated based on spatial stability. This coefficient is implemented through linear mapping or a piecewise function, converting the stability index into a reliability value between 0 and 1. Person / objects with highly stable spatial positions receive a reliability coefficient close to 1, while objects with drastic positional changes have a reliability coefficient close to 0.

[0134] After obtaining the persistence weighting coefficient, the speech energy stability feature in the persistence evaluation index is further processed. Speech energy stability reflects the degree of fluctuation of audio signal energy in the time dimension; stable energy output usually corresponds to clear and coherent speech. The ratio of the standard deviation to the mean of speech energy is calculated as a measure of energy stability; the smaller the ratio, the more stable the energy. Speech energy stability and persistence weighting coefficient are fused, and a weighted product is used to obtain the persistence score. In this fusion process, the persistence weighting coefficient acts as a regulating factor, amplifying or suppressing speech energy stability, so that audio streams with sufficient duration and stable energy obtain higher persistence scores.

[0135] Statistical analysis was performed on the speaking frequency of each character within a historical time window in the state transition probability distribution to obtain the speaking frequency distribution. Speaking frequency is calculated as the cumulative number of times a specific character has acted as the main speaker within a past time period, reflecting the historical distribution of speaking power. Speaking frequency is fused with a state transition penalty factor, and a weighted modulation method is used to obtain a historical state score. In the fusion strategy, for characters with moderate speaking frequency, the state transition penalty factor is close to 1, and the historical state score is mainly determined by speaking frequency; for characters with excessively high speaking frequency, leading to a decrease in the state transition penalty factor, the historical state score decreases accordingly, thus preventing the same character from dominating the main display screen for an extended period.

[0136] After obtaining the persistence score, historical state score, and spatial credibility coefficient, a weighted fusion is performed using state transition constraint rules. These rules define the weight configuration scheme for each scoring dimension and dynamically adjust the weight parameters based on the characteristics of the meeting scenario. For formal presentation scenarios, the weight of the persistence score is increased, making it easier to identify speakers who speak continuously for extended periods as the main speaker. For interactive discussion scenarios, the weight of the historical state score is appropriately increased to avoid excessively frequent screen transitions. For multi-person collaboration scenarios, the weight of the spatial credibility coefficient is increased to ensure that speakers with stable positions in the screen are prioritized for display. The weighted fusion uses a linear weighted summation method, combining the three scoring dimensions according to preset weight coefficients, with the sum of the weight coefficients normalized to 1. Specifically, the initial weighting coefficients are first determined based on the type of meeting scenario: for formal presentation scenarios, the weighting coefficient for persistence score is set to 0.5, the weighting coefficient for historical status score is set to 0.2, and the weighting coefficient for spatial credibility coefficient is set to 0.3; for interactive discussion scenarios, the weighting coefficient for persistence score is set to 0.4, the weighting coefficient for historical status score is set to 0.3, and the weighting coefficient for spatial credibility coefficient is set to 0.3; for multi-person collaboration scenarios, the weighting coefficient for persistence score is set to 0.4, the weighting coefficient for historical status score is set to 0.2, and the weighting coefficient for spatial credibility coefficient is set to 0.4. These initial weighting coefficients are based on statistical analysis results from a large number of actual meeting scenarios. By collecting data on the accuracy of speaker switching and user experience ratings in different types of meetings, machine learning methods are used to train and obtain the optimal weighting configuration for each scenario.

[0137] In practical applications, the weighting coefficients can be fine-tuned based on the real-time detected dynamic features of the meeting. The fine-tuning process uses a linear interpolation method: when a change in the meeting scene features is detected, the similarity between the current scene and each standard scene type is calculated. This similarity is used as the interpolation coefficient, and the weighting coefficients of each standard scene are weighted and averaged to obtain the dynamically adjusted weighting coefficients. To ensure that the sum of the weighting coefficients is always 1, after the interpolation calculation is completed, the three weighting coefficients are normalized by dividing each weighting coefficient by the sum of the three weighting coefficients. Specifically, the persistence score is multiplied by the dynamically adjusted persistence score weighting coefficient, the historical state score is multiplied by the dynamically adjusted historical state score weighting coefficient, and the spatial credibility coefficient is multiplied by the dynamically adjusted spatial credibility coefficient weighting coefficient. The sum of these three weighted results yields the comprehensive score.

[0138] After the comprehensive score is calculated, the candidates are ranked, and the candidate with the highest comprehensive score is identified as the main speaker at that moment. To avoid frequent switching when scores are close, a score difference threshold is set. A screen switch is only performed when the score difference between the new highest-scoring candidate and the current main speaker exceeds this threshold. The score difference threshold is set according to the meeting's stability requirements, typically between 0.1 and 0.2. This threshold is dynamically adjusted based on the current screen stability: if the number of switches exceeds 6 in the last minute, the threshold is increased from 0.15 to 0.2 to enhance stability; if no switch occurs in the last minute and there is significant new speaking activity, the threshold is decreased from 0.15 to 0.1 to improve response sensitivity and ensure clear decision-making basis for screen switching. Simultaneously, the comprehensive score is smoothed over multiple consecutive time steps, using moving average or exponential smoothing methods to reduce the impact of instantaneous fluctuations, making the speaker identification results more stable and reliable. This comprehensive scoring mechanism, through the synergistic constraints of multi-dimensional features, effectively improves the accuracy and robustness of speaker tracking, providing reliable decision support for intelligent screen switching in live conferences.

[0139] Based on the speaker identification information, the main display video stream of the live conference is switched to the video stream corresponding to the current speaker object, and the state transition constraint rules are adjusted based on the playback duration and switching frequency of the switched video stream, including:

[0140] Based on the speaker identification information, the video stream corresponding to the current speaker object is identified, and the main display video stream of the live conference is switched from the currently playing video stream to the video stream corresponding to the current speaker object.

[0141] Record the start time of the video stream corresponding to the current speaker object as the main display video stream, and record the end time of the next video stream switch. Calculate the time difference between the start time and the end time to obtain the playback duration of the switched video stream.

[0142] The number of times the main display video stream is switched is counted within a preset statistical time window, and the switching frequency is calculated based on the number of switches and the duration of the preset statistical time window.

[0143] Based on the playback duration of the switched video stream and the switching frequency, rule adjustment parameters are determined. Based on the rule adjustment parameters, the persistence evaluation index and state transition penalty factor in the state transition constraint rules are adjusted to obtain the updated state transition constraint rules.

[0144] In a live conferencing system, after identifying the current speaker, the corresponding video stream needs to be presented to the audience as the main display video stream. Based on the speaker's identification information, the video stream containing the current speaker is first identified from multiple video streams. This identification process is achieved by matching the stream identifier carried in the speaker's identification information. The stream identifier can be in the form of a numerical number, a unique identification code, or a network address. After identification, a switching command is sent to the video stream distribution control module. This command includes the identification information of the target video stream and switching priority parameters. Upon receiving the switching command, the video stream distribution control module interrupts the output of the current main display video stream and forwards the target video stream data in the buffer to the main display channel. To avoid abrupt screen transitions during the switching process, a fade-in / fade-out effect or a brief transition animation can be used during the transition, with the transition duration typically set to 0.2 to 0.5 seconds.

[0145] After the video stream switch is completed, detailed playback duration information needs to be recorded for subsequent rule adjustments. Specifically, at the moment the switch operation is completed, the current timestamp is obtained as the playback start time, and this timestamp is associated with the current speaker's identifier information and stored in the playback record data structure. This data structure can be in the form of a linked list or a circular queue, with each record node containing fields such as speaker identifier, video stream identifier, playback start time, and playback status flag. When the next speaker change event is detected, the timestamp is obtained again as the playback end time, and the playback end time field in the previous playback record is updated. By calculating the difference between the playback end time and the playback start time, the actual playback duration of this video stream as the main display video stream is obtained. This duration is in seconds, and rounded to one decimal place is sufficient to meet the precision requirements.

[0146] Simultaneously, it's necessary to statistically analyze the video stream switching frequency within a certain time range to assess whether the current speech tracking strategy is overly sensitive. To this end, a preset statistical time window is set, the duration of which is configured according to the characteristics of the meeting scenario, typically ranging from 60 to 300 seconds. Within this time window, a switching event counter is maintained; the counter value increments by 1 each time a main display video stream switching operation occurs. When the time window ends, the switching frequency is calculated by dividing the number of switching events by the time window duration, measured in times per second. For example, if 8 switching events occur within a 120-second time window, the switching frequency is 0.067 times per second, meaning a switching occurs approximately every 15 seconds. To maintain the timeliness of the statistical data, a sliding window mechanism can be used, updating the statistical results at regular intervals (e.g., 10 seconds), removing historical switching events outside the window, and including newly occurring switching events.

[0147] After obtaining the playback duration and switching frequency of the switched video stream, the rule adjustment parameters are determined based on these two indicators. The determination of the rule adjustment parameters follows this logic: When the playback duration is generally short (e.g., average playback duration less than 5 seconds) and the switching frequency is high (e.g., more than 0.1 times per second), it indicates that the current state transition constraint rules are too lenient, leading to excessive sensitivity to brief audio fluctuations and frequent screen switching affecting the viewing experience. In this case, the rule adjustment parameters should be increased to make the constraint rules more stringent. Conversely, when the playback duration is generally long (e.g., average playback duration more than 30 seconds) and the switching frequency is low (e.g., less than 0.02 times per second), it indicates that the constraint rules are too conservative, leading to slow response to changes in the speaker. In this case, the rule adjustment parameters should be decreased to make the constraint rules more flexible.

[0148] The specific parameter determination method can use a piecewise mapping function. First, calculate the average playback duration of the most recent switching events (e.g., the most recent 10 events), denoted as... Set two playback duration thresholds: a short duration threshold and a long duration threshold. Take 5 seconds, long duration threshold Take 30 seconds. At that time, the playback duration evaluation factor The value is 1.5; when hour, The value is 0.5; when hour, The value is set to 1.0. Similarly, two switching frequency thresholds are set: a high-frequency threshold... Set the frequency to 0.1 times per second, a low-frequency threshold. Take 0.02 times per second. When switching frequency At that time, frequency evaluation factor The value is 1.5; when hour, The value is 0.5; when hour, Value: 1.0. Rule adjustment parameter. By combining two evaluation factors, a weighted average method can be used. The weighting coefficients can be adjusted according to actual application needs.

[0149] After determining the rule adjustment parameters, the key indicators in the state transition constraint rules are adjusted based on these parameters. The state transition constraint rules mainly consist of two core components: a persistence evaluation indicator and a state transition penalty factor. The persistence evaluation indicator measures whether the vocalization behavior in an audio stream has sufficient duration, avoiding misjudging brief coughs, laughter, or other informal remarks as speaker switching. This indicator typically sets a minimum duration threshold, denoted as... The initial value can be set to 2 seconds. When adjusting, [the value will be adjusted]. Multiply by the rule to adjust parameters This yields the updated minimum duration threshold. When the value is greater than 1, the minimum duration threshold increases, requiring the sound to last longer before triggering a switch, thus reducing the switching frequency; when When the value is no greater than 1, the minimum duration threshold decreases, resulting in a more agile response to changes in the speaker.

[0150] The state transition penalty factor is used to control the difficulty of speaker state transitions, avoiding screen flickering when multiple speakers take turns speaking. This factor is represented by a penalty weight value, denoted as [value missing]. The initial value can be set to 0.3. When determining whether to switch speakers, the penalty is first calculated using the following formula: ,in The overall score for the current speaker. The overall score for the new speaker candidate. Conditions to be met: A switch can only be triggered if the overall score of the new candidate exceeds that of the current speaker. The state transition penalty factor is multiplied by 1.5. During the dynamic adjustment process, the state transition penalty factor is... Multiply by the rule to adjust parameters The updated state transition penalty factor is obtained: After the update, the formula for calculating the penalty item will change accordingly: The switching condition becomes: When the rule adjusts the parameters When it is greater than 1, the updated state transition penalty factor Increasing the penalty increases the difficulty of switching speakers, which helps stabilize the current speaker's state; when the rule parameters are adjusted... When the value is not greater than 1, the updated state transition penalty factor The penalty is reduced, making it easier to accept new speaker candidates.

[0151] The adjusted state transition constraints are immediately applied to subsequent speaker tracking, continuously monitoring the adjustment effect. If the adjusted switching frequency and playback duration metrics tend to be within a reasonable range, the current rule parameters are maintained; if the expected effect is still not achieved, parameter fine-tuning continues in the next statistical cycle, forming an adaptive optimization closed-loop mechanism. This dynamic adjustment strategy adapts to the characteristics of different meeting scenarios, achieving an optimal balance between image stability and responsiveness, thus improving the viewing experience of live meetings and the accuracy of speaker tracking.

[0152] A second aspect of this invention provides an AI-based speaker tracking system for live conferences, comprising:

[0153] The feature extraction unit is used to acquire multiple audio streams and multiple video streams in a live conference scenario, perform acoustic feature extraction on the multiple audio streams to obtain time-frequency domain feature representations of each audio stream, and perform visual feature extraction on the multiple video streams to obtain spatial location features and dynamic behavior features of people and objects in each video stream.

[0154] The alignment analysis unit is used to determine the correspondence between the audio stream and the video stream based on the time-frequency domain feature representation and the dynamic behavior features, to obtain the aligned multimodal feature sequence, and to determine the person object in the video stream corresponding to the audio stream that has vocal behavior at the current moment based on the aligned multimodal feature sequence, so as to obtain a candidate speaker set.

[0155] The speaker identification unit is used to determine the current speaker object and generate speaker identification information based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking status information of each person object in the speaker candidate set, through state transition constraint rules.

[0156] The video control unit is used to control the main display video stream of the live conference to switch to the video stream corresponding to the current speaker object according to the speaker identification information, and to adjust the state transition constraint rules based on the playback duration and switching frequency of the switched video stream.

[0157] A third aspect of the present invention provides an electronic device, comprising:

[0158] processor;

[0159] Memory used to store processor-executable instructions;

[0160] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0161] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0162] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI-based speaker tracking method for live-streamed conferences, characterized in that, include: The system acquires multiple audio streams and multiple video streams in a live conference scenario, performs acoustic feature extraction on the multiple audio streams to obtain time-frequency domain feature representations of each audio stream, and performs visual feature extraction on the multiple video streams to obtain spatial location features and dynamic behavior features of people and objects in each video stream. Based on the time-frequency domain feature representation and the dynamic behavior features, the correspondence between the audio stream and the video stream is determined, and an aligned multimodal feature sequence is obtained. Based on the aligned multimodal feature sequence, the person in the video stream corresponding to the audio stream that has vocalization behavior at the current moment is determined, and a candidate speaker set is obtained. Based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking status information of each person in the candidate speaker set, the current speaker is determined through state transition constraint rules, and speaker identification information is generated. Based on the speaker identification information, the main display video stream of the live conference is switched to the video stream corresponding to the current speaker object, and the state transition constraint rules are adjusted based on the playback duration and switching frequency of the switched video stream.

2. The method according to claim 1, characterized in that, This involves acquiring multiple audio and video streams from a live conference scenario, performing acoustic feature extraction on the audio streams to obtain time-frequency domain feature representations for each audio stream, and performing visual feature extraction on the video streams to obtain spatial location and dynamic behavior features of objects in each video stream, including: Each audio segment in the multi-channel audio stream undergoes time-frequency transformation processing to extract frequency domain and time domain representation features of the audio segment. The time-frequency domain feature representation is then generated by jointly processing the frequency domain and time domain representation features. The multiple video streams are processed to detect human objects in the images, and the facial region coordinates of each human object are extracted to obtain the spatial region features of the human object; For each character object, a deformation feature sequence of the lip region is extracted. After encoding the deformation feature sequence, a lip movement representation vector and lip movement periodicity features are obtained. Based on the lip movement representation vector, and combined with the head posture change information and facial activation region distribution information of each person, a behavior association matrix is ​​determined; The dynamic behavioral features are generated by modulating the lip movement representation vector based on the behavioral correlation matrix and the periodic features of lip movement.

3. The method according to claim 1, characterized in that, Based on the time-frequency domain feature representation and the dynamic behavior features, the correspondence between the audio stream and the video stream is determined, resulting in an aligned multimodal feature sequence. Then, based on the aligned multimodal feature sequence, the person in the video stream corresponding to the audio stream exhibiting vocal behavior at the current moment is determined, resulting in a candidate speaker set, including: Extract the temporal envelope curve of the energy change pattern in the time-frequency domain feature representation and the amplitude change curve of the lip movement feature in the dynamic behavior feature, calculate the temporal similarity between the temporal envelope curve and the amplitude change curve, and obtain a preliminary matching score between the audio stream and the video stream; Based on the preliminary matching score, audio streams and video streams that meet the similarity constraints are paired, and the timestamp correspondence between the audio streams and video streams is obtained; Based on the timestamp correspondence, the time-frequency domain feature representation, the spatial region features of the person / object, and the dynamic behavior features are aligned and fused to generate the aligned multimodal feature sequence; Temporal analysis is performed on the aligned multimodal feature sequence to determine the target audio stream whose speech activity segment is identified as active in the time-frequency domain feature representation at the current time, and the target video stream mapped by the target audio stream in the timestamp correspondence. Extract the person objects within the time period corresponding to the voice activity segment identifier from the target video stream, verify the consistency of the person objects' voices based on the spatial region features and dynamic behavior features, and obtain the speaker candidate set.

4. The method according to claim 3, characterized in that, Extract the person objects within the time period corresponding to the speech activity segment identifier from the target video stream, verify the consistency of the person objects' vocalizations based on the spatial region features and dynamic behavior features, and obtain the speaker candidate set, including: Extract all human objects within the time period corresponding to the voice activity segment identifier from the target video stream, and obtain the spatial region features and dynamic behavior features of each human object; Based on the spatial region features of each person object, the spatial position distribution stability of each person object in the target video stream is calculated, and person objects whose spatial position distribution stability satisfies the visibility constraint are selected to obtain the spatial filtering result; For each person in the spatial filtering results, extract the periodic features of lip movements from the dynamic behavior features, and calculate the temporal overlap between the periodic features of lip movements and the time range of the speech activity segment identifier; For each person in the spatial filtering results, the facial activation pattern in the dynamic behavior features is extracted, and a multi-dimensional consistency evaluation index is determined by combining the periodic features of lip movement with the facial activation pattern. Based on the multi-dimensional consistency evaluation index and the temporal overlap, the vocal consistency of each person is determined, and the person whose vocal consistency meets the consistency judgment condition is added to the speaker candidate set.

5. The method according to claim 1, characterized in that, Based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking state information of each individual in the candidate speaker set, the current speaker is determined through state transition constraint rules, and speaker identification information is generated, including: Based on the persistence characteristics represented by the time-frequency domain features, the duration of speech activity and the stability of speech energy of the audio stream corresponding to each speaker in the candidate speaker set are calculated to generate a persistence evaluation index; Based on the historical speaking status information, the speaking frequency and speaking transition pattern of each individual in the speaker candidate set within the historical time window are extracted to determine the state transition probability distribution; Based on the spatial location characteristics of each individual in the candidate speaker set, the spatial distance relationship and spatial location stability between each individual are calculated, and spatial constraints are generated. Based on the continuous evaluation index, the state transition probability distribution, and the spatial constraints, the comprehensive score of each person in the speaker candidate set is determined using the state transition constraint rules; The person with the highest overall score is selected as the current speaker, and the speaker identification information is generated based on the current speaker.

6. The method according to claim 5, characterized in that, Based on the aforementioned continuous evaluation indicators, the aforementioned state transition probability distribution, and the aforementioned spatial constraints, the comprehensive score of each individual in the speaker candidate set is determined using the aforementioned state transition constraint rules, including: Extract the duration of speech activity of each person in the audio stream corresponding to the duration of speech activity from the duration of speech activity, and calculate the duration of duration of speech activity based on the ratio of the duration of speech activity to the preset duration of duration of duration. Extract the historical speech transition frequency of each character in the state transition probability distribution, and calculate the state transition penalty factor based on the historical speech transition frequency; Extract the spatial positional stability of each character object under the spatial constraints, and calculate the spatial reliability coefficient based on the spatial positional stability. The persistence score is obtained by fusing the speech energy stability of each individual in the persistence evaluation index with the persistence weight coefficient, and the historical state score is obtained by fusing the speaking frequency of each individual in the state transition probability distribution within the historical time window with the state transition penalty factor. Based on the state transition constraint rules, the persistence score, the historical state score, and the spatial credibility coefficient are weighted and fused to obtain the comprehensive score of each person in the speaker candidate set.

7. The method according to claim 1, characterized in that, Based on the speaker identification information, the main display video stream of the live conference is switched to the video stream corresponding to the current speaker object, and the state transition constraint rules are adjusted based on the playback duration and switching frequency of the switched video stream, including: Based on the speaker identification information, the video stream corresponding to the current speaker object is identified, and the main display video stream of the live conference is switched from the currently playing video stream to the video stream corresponding to the current speaker object. Record the start time of the video stream corresponding to the current speaker object as the main display video stream, and record the end time of the next video stream switch. Calculate the time difference between the start time and the end time to obtain the playback duration of the switched video stream. The number of times the main display video stream is switched is counted within a preset statistical time window, and the switching frequency is calculated based on the number of switches and the duration of the preset statistical time window. Based on the playback duration of the switched video stream and the switching frequency, rule adjustment parameters are determined. Based on the rule adjustment parameters, the persistence evaluation index and state transition penalty factor in the state transition constraint rules are adjusted to obtain the updated state transition constraint rules.

8. An AI-based speaker tracking system for live-streamed conferences, used to implement the method as described in any one of claims 1-7, characterized in that, include: The feature extraction unit is used to acquire multiple audio streams and multiple video streams in a live conference scenario, perform acoustic feature extraction on the multiple audio streams to obtain time-frequency domain feature representations of each audio stream, and perform visual feature extraction on the multiple video streams to obtain spatial location features and dynamic behavior features of people and objects in each video stream. The alignment analysis unit is used to determine the correspondence between the audio stream and the video stream based on the time-frequency domain feature representation and the dynamic behavior features, to obtain the aligned multimodal feature sequence, and to determine the person object in the video stream corresponding to the audio stream that has vocal behavior at the current moment based on the aligned multimodal feature sequence, so as to obtain a candidate speaker set. The speaker identification unit is used to determine the current speaker object and generate speaker identification information based on the spatial location features, time-frequency domain feature representation persistence features, and historical speaking status information of each person object in the speaker candidate set, through state transition constraint rules. The video control unit is used to control the main display video stream of the live conference to switch to the video stream corresponding to the current speaker object according to the speaker identification information, and to adjust the state transition constraint rules based on the playback duration and switching frequency of the switched video stream.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images