A Method and System for Intelligent Voice Conferencing Behavior Analysis Based on Multi-Source Perception
By constructing a multi-source collaborative attribution graph and a conference collaboration graph structure, and combining multi-dimensional path similarity comparison and structural coupling degree function, the problem that existing systems cannot recognize non-voice input behavior is solved, and a comprehensive analysis and reliable assessment of conference behavior is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIAN XIANGE ACOUSTIC ELECTRONIC CO LTD
- Filing Date
- 2025-11-06
- Publication Date
- 2026-06-30
AI Technical Summary
Existing intelligent meeting analysis systems cannot effectively identify non-voice input behaviors, resulting in silent participation behaviors not being included in the analysis, and in multi-microphone environments, errors in speaker attribution are prone to occur.
By collecting audio signals, collaborative behavior data, and terminal device data, a multi-source collaborative attribution graph structure and a meeting collaboration graph structure are constructed. By combining multi-dimensional path similarity comparison and structural coupling degree function, explicit speaking and implicit collaborative participation are identified and quantified.
It achieves unified modeling of speech and non-speech behaviors within the same temporal framework, improving the credibility and interpretability of meeting behavior analysis, accurately identifying silent participation behavior, and correcting speaking attribution errors.
Smart Images
Figure CN121462328B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-source data processing and analysis technology, specifically to a method and system for intelligent voice conferencing behavior analysis based on multi-source perception. Background Technology
[0002] In existing intelligent meeting analysis systems, speech signals during meetings are typically extracted and segmented using speech recognition and sound source localization algorithms. The speaker's identity is then determined based on the audio signal energy threshold, waveform characteristics, and acoustic model, enabling automatic generation of meeting minutes or statistical analysis of speaking behavior. However, these systems generally rely on single-modal speech input and lack a unified structured processing mechanism for non-voice data such as collaborative behaviors, document editing operations, and terminal interaction records during meetings. This results in meeting behavior analysis primarily remaining at the acoustic dimension, lacking multi-source data support for assessing the credibility and participation of participants' behavior. Consequently, in complex meeting scenarios, it is difficult to comprehensively reflect the participants' true behavioral patterns and their contribution to meeting decisions.
[0003] In actual meeting scenarios, there are two problems that make it difficult to perceive and judge meeting behaviors: First, some participants may not speak, but they may still have a substantial impact on meeting decisions through document editing, parameter adjustment, screen operation, etc. Existing systems lack a mechanism for recognizing and quantifying non-voice input behaviors, and therefore cannot include such silent participation behaviors in meeting behavior analysis, resulting in inconsistencies between the behavior records output by the system and the actual decision-making participation. Second, when multiple participants share the same microphone or are in adjacent spatial positions, sound source signals are prone to overlap or energy crossover, causing the system to mistakenly identify the voice of others as the speaker of the person currently using the microphone during the sound source recognition stage, thus resulting in errors in attribution of speaking rights and deviations in responsibility recording. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for intelligent voice conference behavior analysis based on multi-source perception, so as to solve the problem of difficulty in perceiving and judging the two conference behaviors mentioned in the background art.
[0005] To achieve the above objectives, the technical solution of the present invention is: an intelligent voice conference behavior analysis method based on multi-source perception, comprising:
[0006] S1. Collect audio signals, collaborative behavior data and terminal device data during the meeting, and use the meeting timestamp dynamic window segmentation mechanism to construct a time slice index data structure, voice spatial positioning sequence and terminal activation record sequence;
[0007] S2. Extract the sound source direction and intensity vector of the audio signal, construct the activation state correspondence matrix between the conference speakers and the device by combining terminal device data, construct a multi-source collaborative attribution graph structure based on the speech spatial localization sequence and the terminal activation record sequence, and use the multi-dimensional path similarity comparison method in the multi-source collaborative attribution graph structure to determine the optimal mapping relationship between the audio signal and the participants, and obtain the explicit speech attribution result.
[0008] S3. Identify non-voice input behaviors based on collaborative behavior data and construct a meeting collaboration graph structure. Perform semantic consistency comparison between non-voice input behaviors and time slice index data structure. Use the structural coupling degree function to calculate the structural contribution strength of non-voice input behaviors to the meeting decision content. Calculate the non-voice contribution strength sequence based on the structural contribution strength and the weight of each node in the meeting collaboration graph to obtain the implicit behavior identification result.
[0009] S4. Construct a meeting behavior credibility matrix based on explicit speech attribution results and implicit behavior recognition results. Perform temporal consistency verification on the meeting behavior credibility matrix based on the participant identity. Eliminate duplicate mapping records of meeting decision content with multiple mappings of participant identity in the same time slice to obtain the meeting behavior credibility recognition results.
[0010] Preferably, in S1, the meeting timestamp dynamic window segmentation mechanism refers to a control function that dynamically determines the window duration based on the meeting timeline as an index reference and the audio energy change rate and the number of collaborative behavior triggers. When the window duration reaches a preset time threshold and the number of collaborative behavior triggers reaches a cumulative threshold, the window duration is updated to generate a new time slice, which is used to segment and synchronize the audio signal, collaborative behavior data and terminal device data in the meeting, and to construct a time slice index data structure, a voice spatial positioning sequence and a terminal activation record sequence.
[0011] Preferably, in S2, the activation state correspondence matrix refers to a two-dimensional matrix structure established with the time slice index as the row index and the terminal device and audio signal features as the column index. The values of the activation state correspondence matrix are the microphone activation state of the corresponding terminal device in the time slice, the level change amount, and the correlation strength between them and the peak value of the audio signal energy.
[0012] The method for constructing the activation state correspondence matrix is as follows: read terminal device data, calculate the average audio input power and level change rate of each terminal device in the corresponding time slice, and perform normalization matching with the speech spatial positioning sequence to obtain the value of the activation state correspondence matrix, thereby constructing the activation state correspondence matrix.
[0013] Preferably, in S2, the multi-source collaborative attribution graph structure refers to a directed weighted graph structure with participant identity and time slice index as nodes and the response coupling relationship between audio signal and terminal device as edges, used to describe the attribution path relationship between audio signal and participant identity;
[0014] The method for constructing the multi-source collaborative attribution graph structure is as follows: Based on the activation state correspondence matrix, the correlation strength between each terminal device and the direction of the sound source is extracted as the basis for edge weight calculation. The edge weight is defined as a linear combination of the sound source azimuth angle difference and the terminal device response delay. A connection edge is established between the identity of each participant and the time slice index within the corresponding time slice to form the multi-source collaborative attribution graph structure.
[0015] Preferably, in S2, the multi-dimensional path similarity comparison method refers to calculating the path similarity from each participant's identity to the time slice index path in the multi-source collaborative attribution graph structure, using sound source direction features, sound intensity vector features, and terminal device activation delay features as comparison dimensions, and using a weighted cosine similarity function to obtain the path matching degree.
[0016] The optimal mapping relationship refers to selecting the path with the highest path matching degree and edge weight consistency that meets the threshold constraint among all paths from participant identity to time slice index, as the unique mapping path between audio signal and participant identity, which is the optimal mapping relationship.
[0017] The explicit speech attribution result refers to the paired records of the optimal mapping relationship, which are used to determine the explicit attribution result between the speaker's identity and the speech content during the meeting behavior analysis process.
[0018] Preferably, in S3, the meeting collaboration graph structure refers to a directed weighted graph structure with participant identities and collaborative behavior events as nodes and collaborative relationships and event dependencies as edges, used to describe the operation and interaction paths between participants and between participants and meeting content in the meeting;
[0019] The method for constructing the meeting collaboration graph structure is as follows: Based on the collaborative behavior event data, establish connection edges between participant identity nodes and corresponding collaborative behavior events, take the order of operation time as the direction attribute of the edge, and take the similarity of operation content and the normalized reciprocal of the time interval between operations as the edge weight; perform edge aggregation processing on event nodes that share the same operation goal among different participant identities to obtain the meeting collaboration graph structure.
[0020] Preferably, in S3, the structural coupling function is a function that calculates the structural correlation strength between non-voice input behavior and meeting decision content based on the meeting collaboration graph structure, and is used to calculate the structural contribution strength of non-voice input behavior to meeting decision content;
[0021] The calculation process of the structural coupling degree function is as follows: calculate the semantic similarity between the feature vectors of collaborative behavior event nodes and meeting decision content, calculate the path decay factor in combination with the shortest path length in the meeting collaboration graph structure, and use the weighted product of the path decay factor and semantic similarity as the structural coupling degree value of the collaborative behavior event node. By calculating the structural coupling degree values of all collaborative behavior event nodes, the structural contribution strength of non-voice input behavior to the meeting decision content is obtained.
[0022] Preferably, in S3, the non-speech contribution intensity sequence refers to the structural contribution intensity sequence of non-speech input behavior in each time slice, recorded in the order of time slice index.
[0023] The calculation method for the non-voice contribution intensity sequence is as follows: based on the structural contribution intensity value of each collaborative behavior event node, the weighted sum is calculated according to the edge weight between the participant identity node and the collaborative event node in the meeting collaboration graph, and the results are arranged in time slice order to form the non-voice contribution intensity sequence.
[0024] The implicit behavior recognition result refers to determining, based on the time slice range in the non-voice contribution intensity sequence that continuously exceeds a preset intensity threshold, whether the corresponding participant has a collaborative behavior with high structural contribution intensity within that time slice, and marking the collaborative behavior as an implicit behavior recognition result.
[0025] Preferably, in S4, the meeting behavior credibility matrix refers to a two-dimensional matrix structure established with the participant's identity as the row index and the meeting time slice as the column index. The value of the meeting behavior credibility matrix is jointly calculated by the explicit speech attribution result and the implicit behavior identification result, representing the comprehensive credibility of the participant's speaking behavior and collaborative behavior within a specific time slice.
[0026] The method for constructing the meeting behavior credibility matrix is as follows: within each time slice, the path matching degree of the explicit speech attribution result and the non-voice contribution intensity value in the implicit behavior recognition result are weighted and fused to calculate the behavior credibility value of the participant in that time slice, and written into the matrix unit to form the meeting behavior credibility matrix.
[0027] On the other hand, the present invention provides an intelligent voice conference behavior analysis system based on multi-source perception, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the aforementioned intelligent voice conference behavior analysis method based on multi-source perception.
[0028] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects:
[0029] 1. In this invention, based on a multi-source perception fusion mechanism, audio data, collaborative behavior data and terminal device data during the meeting can be collected and associated simultaneously. Through dynamic time-slice segmentation and multi-dimensional path similarity analysis, unified modeling of voice behavior and non-voice behavior is achieved, enabling the system to identify the speaker's identity and operational contribution within the same temporal framework, thereby avoiding the one-sidedness of behavior recognition caused by traditional meeting systems relying solely on voice signals.
[0030] 2. In this invention, by constructing a multi-source collaborative attribution graph structure and a meeting collaboration graph structure and introducing a behavior credibility matrix calculation mechanism, a comprehensive evaluation of participants' explicit speaking and implicit collaborative participation can be achieved. This can quantitatively identify silent participation behavior and correct speaking attribution errors in multi-microphone environments, thereby improving the credibility and interpretability of meeting behavior analysis results. Attached Figure Description
[0031] Figure 1 This is a flowchart of an embodiment of the present invention. Detailed Implementation
[0032] Example 1, as Figure 1 As shown, the specific implementation steps of the intelligent voice conference behavior analysis method based on multi-source perception proposed in this invention are as follows:
[0033] S1. Collect audio signals, collaborative behavior data and terminal device data during the meeting, and use the meeting timestamp dynamic window segmentation mechanism to construct a time slice index data structure, voice spatial positioning sequence and terminal activation record sequence;
[0034] S2. Extract the sound source direction and intensity vector of the audio signal, construct the activation state correspondence matrix between the conference speakers and the device by combining terminal device data, construct a multi-source collaborative attribution graph structure based on the speech spatial localization sequence and the terminal activation record sequence, and use the multi-dimensional path similarity comparison method in the multi-source collaborative attribution graph structure to determine the optimal mapping relationship between the audio signal and the participants, and obtain the explicit speech attribution result.
[0035] S3. Identify non-voice input behaviors based on collaborative behavior data and construct a meeting collaboration graph structure. Perform semantic consistency comparison between non-voice input behaviors and time slice index data structure. Use the structural coupling degree function to calculate the structural contribution strength of non-voice input behaviors to the meeting decision content. Calculate the non-voice contribution strength sequence based on the structural contribution strength and the weight of each node in the meeting collaboration graph to obtain the implicit behavior identification result.
[0036] S4. Construct a meeting behavior credibility matrix based on explicit speech attribution results and implicit behavior recognition results. Perform temporal consistency verification on the meeting behavior credibility matrix based on the participant identity. Eliminate duplicate mapping records of meeting decision content with multiple mappings of participant identity in the same time slice to obtain the meeting behavior credibility recognition results.
[0037] In this embodiment S1, the meeting timestamp dynamic window segmentation mechanism refers to a control function that uses the meeting timeline as an index reference and calculates and dynamically determines the window duration based on the audio energy change rate and the number of collaborative behavior triggers. When the window duration reaches a preset time threshold and the number of collaborative behavior triggers reaches a cumulative threshold, the window duration is updated to generate a new time slice. This is used to segment and synchronize the audio signals, collaborative behavior data, and terminal device data in the meeting, and to construct a time slice index data structure, a voice spatial positioning sequence, and a terminal activation record sequence.
[0038] In this embodiment S1, the audio signal refers to the multi-channel acoustic signal acquired in real time by the microphone array set in the conference acquisition terminal. The microphone array acquires the original speech waveform data in the conference space at a fixed sampling frequency, and records the timestamp, sound pressure level, and signal arrival time difference simultaneously during the acquisition process, which are used to calculate the sound source direction and sound intensity vector in subsequent steps. The collaborative behavior data refers to the operation records generated by the participants through the conference terminal during the conference, including editing of shared documents, parameter adjustment during screen sharing, and text input in the instant messaging channel. The system acquires the above operation behaviors in real time in the form of event stream through the collaborative platform interface and records the operation time, operation object, and operation parameters. The terminal device data refers to the operating status information of the participating terminal, including terminal identification, microphone on / off status, volume level, speaker output status, and the binding relationship between the terminal and the participant's identity. The terminal device data is acquired in real time by the device status monitoring module of the conference system and stored synchronously with the system clock.
[0039] In this embodiment S1, the audio energy change rate refers to the degree of change in the energy amplitude of the conference audio signal within a continuous sampling period, used to reflect the fluctuation characteristics of speech activities in the conference. The system performs short-time energy calculation on the multi-channel audio signal at fixed time intervals, obtains the average sound pressure level squared value within each sampling window, and calculates the energy change rate between adjacent sampling windows. When the energy change rate exceeds the set energy fluctuation threshold, it is determined that there is a new speech activity or the start of a speech segment in that time interval. The audio energy change rate is used to dynamically reflect the activity level of the speech signal during the conference, providing a real-time reference for adjusting the window duration. The number of times collaborative behavior is triggered refers to the number of times during the conference. In the collaborative platform, the number of collaborative events detected per unit time is used to characterize the activity level of non-voice operations. The system monitors event trigger records (including document editing, parameter modification, interface switching, message input, etc.) in the collaborative behavior data stream, counts the cumulative number of event triggers within the current window time period according to the meeting timeline, and resets the cumulative count to zero when the time slice is updated. When the number of collaborative behavior triggers reaches the preset cumulative threshold, it indicates that non-voice behaviors are intensive within that time period, and the system automatically triggers the time window update operation to ensure that the dynamic segmentation mechanism can respond to changes in voice and collaborative operations simultaneously, thereby achieving adaptive synchronous segmentation of multi-source data.
[0040] In this embodiment S1, the time-slice index data structure refers to a multi-source data alignment structure established with the meeting timestamp as the index benchmark. The structure uses the time-slice number as the primary key and records audio signal segments, collaborative behavior events, and terminal device status records within the corresponding time slice to ensure the synchronization consistency of different types of data in the time dimension. The speech spatial positioning sequence refers to an ordered sequence composed of the sound source direction angle and intensity vector extracted from the audio signal. The sequence is used to describe the dynamic distribution of the sound source in the meeting space and serves as the spatial input parameter for subsequent explicit speech attribution. The terminal activation record sequence refers to an ordered record of the microphone activation status, level changes, and corresponding participant identity binding information of each participating terminal recorded in each time slice of the time-slice index data structure to reflect the speaking status of each participating terminal during the meeting.
[0041] In this embodiment S1, the specific method for constructing the time slice index data structure, voice spatial positioning sequence, and terminal activation record sequence using the meeting timestamp dynamic window segmentation mechanism is as follows: Using the meeting timeline as the index reference, the system monitors the rate of change of audio signal energy and the number of collaborative behavior event triggers in real time. The system calculates the dynamic window duration Δt based on the amplitude change of the energy rate of change and the event trigger density, and uses the dynamic window duration as the time slice boundary control parameter. When the window duration reaches a preset time threshold or the number of collaborative behavior event triggers reaches a cumulative threshold, a time slice update operation is triggered, generating a new time slice index number. During each window update, the audio signal, collaborative behavior data, and terminal device data within the new time slice are written into the corresponding storage area of the time slice index data structure. Simultaneously, the arrival time difference and sound pressure intensity distribution of each channel's audio signal within the time slice are calculated to form a voice spatial positioning sequence, and the activation status and identity binding information of each participating terminal are recorded to form a terminal activation record sequence.
[0042] In this embodiment S2, the activation state correspondence matrix refers to a two-dimensional matrix structure established with the time slice index as the row index and the terminal device and audio signal features as the column index. The values of the activation state correspondence matrix are the microphone activation state of the corresponding terminal device in the time slice, the level change amount, and the correlation strength between them and the peak value of the audio signal energy.
[0043] The method for constructing the activation state correspondence matrix is as follows: read terminal device data, calculate the average audio input power and level change rate of each terminal device in the corresponding time slice, and perform normalization matching with the speech spatial positioning sequence to obtain the value of the activation state correspondence matrix, thereby constructing the activation state correspondence matrix.
[0044] In this embodiment S2, the sound source direction of the audio signal refers to the incident azimuth and elevation angles of the speech signal in the microphone array coordinate system within the conference space, used to characterize the directional position of the sound source relative to the geometric center of the array; the intensity vector refers to the vectorized acoustic feature calculated from the sound pressure and particle velocity components in the sampled signals of each channel, used to describe the amplitude and directional distribution of the sound source radiated energy in spatial propagation; the sound source direction and the intensity vector together constitute the spatial features of the audio signal, used to identify the correspondence between the sound source and the participants' positions in subsequent steps; the method for extracting the sound source direction of the audio signal is: using the microphone array signal of the conference acquisition terminal, through adjacent channels... The system calculates the incident angle of the sound wave by taking the time difference of arrival of the signal between time slots, and performs least-squares spatial calculation in combination with the array geometric layout parameters to obtain the azimuth and elevation coordinates of the sound source in each time slot. The method for extracting the sound intensity vector is as follows: perform short-time Fourier transform on the sampled signals of each channel in the microphone array to obtain the frequency domain energy distribution, calculate the product of sound pressure and particle velocity in the same frequency band and perform vectorization processing to obtain the sound intensity vector value in each direction. After extracting the sound source direction and sound intensity vector, the system performs normalization and weighted averaging processing on the results in the same time slot to form the complete spatial features of each record in the speech spatial localization sequence, which are used for classification comparison with the data of the terminal device.
[0045] In this embodiment S2, the multi-source collaborative attribution graph structure refers to a directed weighted graph structure with participant identity and time slice index as nodes and the response coupling relationship between audio signal and terminal device as edges, which is used to describe the attribution path relationship between audio signal and participant identity.
[0046] The method for constructing the multi-source collaborative attribution graph structure is as follows: Based on the activation state correspondence matrix, the correlation strength between each terminal device and the direction of the sound source is extracted as the basis for edge weight calculation. The edge weight is defined as a linear combination of the sound source azimuth angle difference and the terminal device response delay. A connection edge is established between the identity of each participant and the time slice index within the corresponding time slice to form the multi-source collaborative attribution graph structure.
[0047] In this embodiment S2, the nodes of the multi-source collaborative attribution graph structure include two types: participant identity nodes and time slice index nodes. Participant identity nodes are used to identify the binding relationship between participants and terminal devices, and time slice index nodes are used to identify the corresponding audio data segments in the time slice index data structure. Each directed edge points from an identity node to a time slice node and is used to describe the spatial attribution relationship of the participant's identity to the corresponding audio signal within a specific time slice. The edge weights in this graph structure are used to quantitatively represent the consistency strength between the sound source response characteristics of the terminal device and the spatial characteristics of the audio signal, thereby achieving accurate determination of speech attribution in scenarios where multiple participants share a microphone or multiple sound sources overlap.
[0048] In this embodiment S2, the basis for edge weight calculation is the correlation strength between the terminal device and the sound source direction in the active state correspondence matrix. The correlation strength is calculated by the correlation coefficient between the terminal device level change rate and the peak energy of the audio signal in the matrix unit. The system traverses the active state correspondence matrix in each time slice, calculates the above correlation coefficient for each terminal device under the same sound source direction, and performs standardization processing to obtain the correlation strength value between each terminal device and the sound source direction, which serves as the input basis for edge weight calculation. The sound source azimuth angle difference refers to the angle difference between the sound source direction angle and the terminal device spatial azimuth angle in the three-dimensional coordinate system. The terminal device response delay refers to the time interval from the appearance of the sound source signal to the acquisition of the terminal microphone activation signal. The system calculates edge weights based on the quantization results of both. The edge weights are defined as a linear combination of the source azimuth angle difference and the terminal device response delay, used to balance the contribution ratio of spatial direction error and time response error in the edge weights. The purpose of using a linear combination of the source azimuth angle difference and the terminal device response delay as the edge weight definition is that the source azimuth angle difference reflects the consistency of spatial positioning, and the terminal device response delay reflects the responsiveness of time synchronization. The combination of the two can simultaneously quantify the coupling degree between the source and the device in terms of spatial position and time response, so that the edge weights not only reflect the spatial proximity of the source, but also reflect the device's real-time response capability to the spoken signal, thereby ensuring that the multi-source collaborative attribution map structure achieves collaborative alignment of spatial and temporal dimensions during the mapping process.
[0049] In this embodiment S2, the multi-dimensional path similarity comparison method refers to calculating the path similarity from each participant's identity to the time slice index path in the multi-source collaborative attribution graph structure, using sound source direction features, sound intensity vector features, and terminal device activation delay features as comparison dimensions, and using a weighted cosine similarity function to obtain the path matching degree.
[0050] The optimal mapping relationship refers to selecting the path with the highest path matching degree and edge weight consistency that meets the threshold constraint among all paths from participant identity to time slice index, as the unique mapping path between audio signal and participant identity, which is the optimal mapping relationship.
[0051] The explicit speech attribution result refers to the paired records of the optimal mapping relationship, which are used to determine the explicit attribution result between the speaker's identity and the speech content during the meeting behavior analysis process.
[0052] In this embodiment S2, the process of determining the optimal mapping relationship between audio signals and participants using a multi-dimensional path similarity comparison method in a multi-source collaborative attribution graph structure includes: considering the directed connection between any participant identity node and any time slice index node in the multi-source collaborative attribution graph structure as a candidate attribution path, wherein the candidate attribution path consists of a series of directed edges, each carrying a corresponding edge weight; for each candidate attribution path, extracting the sound source direction features, sound intensity vector features, and terminal device activation delay features within the corresponding time slice of the path, wherein the sound source direction... The features are used to characterize the spatial consistency between the azimuth angle of the sound source and the spatial azimuth angle of the participant's corresponding terminal device within the time slice; the sound intensity vector features are used to characterize the amplitude consistency between the sound pressure intensity amplitude distribution of the audio signal within the time slice and the audio input power picked up by the participant's corresponding terminal device; the terminal device activation delay features are used to characterize whether the response timing of the participant's corresponding terminal device to the audio signal is synchronized with the time when the audio signal appears; after extraction, the above three types of features are combined in a fixed order to form the feature vector representation of the candidate attribution path.
[0053] In this embodiment S2, for each candidate destination path, the system calculates path similarity based on the feature vector representation. The path similarity calculation process is as follows: using the cosine of the angle between the feature vector of the candidate destination path and the reference speech feature vector as the basic similarity index, weighting coefficients are set for the sound source direction feature, sound intensity vector feature, and terminal device activation delay feature, respectively. A weighted cosine similarity calculation is performed on the three feature components to obtain the path matching degree of the candidate destination path. The reference speech feature vector refers to the feature vector representation of spatial direction, acoustic energy, and response delay extracted from the audio signal within the current time slice. Through this weighted cosine similarity calculation, the path matching degree is improved. Simultaneously reflecting spatial consistency, energy consistency, and temporal consistency, it provides a unified metric for speech attribution. After obtaining the path matching degree of all candidate attribution paths, the path matching degree is used as the sorting criterion, and the candidate attribution path with the highest matching degree and all edge weights on the path meeting the preset consistency threshold requirement is selected as the optimal mapping relationship. The preset consistency threshold requirement is used to constrain the deviation of the edge weights on each edge of the candidate attribution path in spatial direction difference and time response delay to not exceed the allowable range, so as to avoid false high matching paths caused by local short-term noise interference or instantaneous resonance. The optimal mapping relationship is expressed as the correspondence between the uniquely determined participant identity node and the time slice index node.
[0054] In this embodiment S2, the explicit speech attribution result refers to recording the optimal mapping relationship obtained for each time slice, forming a one-to-one binding relationship between the participant's identity and the audio signal within the time slice, and using the one-to-one binding relationship as the explicit speech attribution result output; the explicit speech attribution result provides a record of speech behavior based on identity confirmation for subsequent steps.
[0055] In this embodiment S3, the meeting collaboration graph structure refers to a directed weighted graph structure with participant identities and collaborative behavior events as nodes and collaborative relationships and event dependencies as edges, used to describe the operation and interaction paths between participants and between participants and meeting content in the meeting;
[0056] The method for constructing the meeting collaboration graph structure is as follows: Based on the collaborative behavior event data, establish connection edges between participant identity nodes and corresponding collaborative behavior events, take the order of operation time as the direction attribute of the edge, and take the similarity of operation content and the normalized reciprocal of the time interval between operations as the edge weight; perform edge aggregation processing on event nodes that share the same operation goal among different participant identities to obtain the meeting collaboration graph structure.
[0057] In this embodiment S3, the nodes of the meeting collaboration graph structure include participant identity nodes and collaboration behavior event nodes. The collaboration relationship refers to the operation triggering relationship between the participant identity node and the collaboration behavior event node executed in the meeting system, which is used to represent the specific operation behavior of a participant during the meeting. The event dependency relationship refers to the operation sequence and logical dependency relationship between different collaboration behavior event nodes, which is used to represent the dependence of one operation event on the triggering or content modification of another event. The operation interaction path in the meeting collaboration graph structure refers to the directed path formed by connecting the participant identity node, collaboration event node, and meeting decision content node through collaboration relationship edges and event dependency relationship edges, which is used to describe the continuous operation process and influence transmission link of a participant on a certain meeting content during the meeting. Operation content similarity refers to the degree of similarity between two collaborative behavior event nodes in terms of operation target or text content. It is calculated by comparing the degree of overlap of target identifiers, text feature vectors, or editing instruction sets involved in the operation events, and is usually quantified using a cosine similarity function. The time interval between operations refers to the difference in timestamps between two consecutive collaborative behavior events, which is used to reflect the temporal tightness of the operation behaviors. The system multiplies the operation content similarity by the normalized reciprocal of the time interval between operations to obtain the edge weight, which is used to measure the correlation strength between collaborative events. When multiple participant identity nodes generate collaborative behaviors on the same meeting target, the system identifies the shared target identifier of these behavior nodes, performs edge aggregation processing on event nodes under the same target, and performs weighted average of the edge weights of similar edges and merges them into a single aggregated edge.
[0058] In this embodiment S3, the structural coupling function refers to the function that calculates the structural correlation strength between non-voice input behavior and meeting decision content based on the meeting collaboration graph structure, and is used to calculate the structural contribution strength of non-voice input behavior to meeting decision content;
[0059] The calculation process of the structural coupling degree function is as follows: calculate the semantic similarity between the feature vectors of collaborative behavior event nodes and meeting decision content, calculate the path decay factor in combination with the shortest path length in the meeting collaboration graph structure, and use the weighted product of the path decay factor and semantic similarity as the structural coupling degree value of the collaborative behavior event node. By calculating the structural coupling degree values of all collaborative behavior event nodes, the structural contribution strength of non-voice input behavior to the meeting decision content is obtained.
[0060] In this embodiment S3, the semantic similarity calculation method between the feature vectors of collaborative behavior event nodes and meeting decision content nodes is as follows: First, the operation text, parameter modification records, or editing instruction sequences in the collaborative behavior event nodes are extracted and converted into semantic vector representations using a word vector model or embedding vector model; second, the decision text, title keywords, and topic embedding vectors in the meeting decision content nodes are extracted and also converted into semantic vector representations; then, the cosine similarity value between the two sets of semantic vectors is calculated to measure the degree of semantic correspondence between the collaborative behavior event and the meeting decision content; when calculating the path decay factor, the system uses the shortest path length between the collaborative behavior event node and the meeting decision content node in the meeting collaboration graph structure as the input parameter and adopts an exponential decay model to describe the influence of the structural propagation distance between the collaborative behavior event and the decision content on the coupling degree; the smaller the path decay factor, the shorter the propagation path and the stronger the structural association; finally, the system performs a weighted product of the semantic similarity and the path decay factor according to the proportional coefficient in the exponential decay model to obtain the structural coupling degree value output by the structural coupling degree function.
[0061] In this embodiment S3, the non-speech contribution intensity sequence refers to the structural contribution intensity sequence of non-speech input behavior in each time slice, recorded in the order of time slice index.
[0062] The calculation method for the non-voice contribution intensity sequence is as follows: based on the structural contribution intensity value of each collaborative behavior event node, the weighted sum is calculated according to the edge weight between the participant identity node and the collaborative event node in the meeting collaboration graph, and the results are arranged in time slice order to form the non-voice contribution intensity sequence.
[0063] The implicit behavior recognition result refers to determining, based on the time slice range in the non-voice contribution intensity sequence that continuously exceeds a preset intensity threshold, whether the corresponding participant has a collaborative behavior with high structural contribution intensity within that time slice, and marking the collaborative behavior as an implicit behavior recognition result.
[0064] In this embodiment S3, the time slice range in which the non-voice contribution intensity sequence continuously exceeds the preset intensity threshold refers to a number of consecutive time slices in the time slice index order, where the corresponding non-voice contribution intensity values are all greater than the preset intensity threshold, indicating that the participant has a continuous high degree of structural correlation in collaborative behavior during this time period. The preset intensity threshold is obtained by the system from the historical samples of meeting data. By calculating the mean and standard deviation of the non-voice contribution intensity values of all participants, the threshold is determined to be in the form of μ+σ, where μ is the average intensity value and σ is the standard deviation, used to distinguish between normal collaborative operations and significantly high correlation operations. The time slice range refers to the set of consecutive time slices generated by the meeting timestamp dynamic window segmentation mechanism. Each time slice has a unique time index number and a fixed duration, used to reflect the discrete interval of the meeting timeline. When the non-voice contribution intensity sequence continuously exceeds the preset intensity threshold within a certain consecutive time slice range, the system determines that the participant has a significant collaborative participation contribution behavior within that time slice range and marks this behavior as a latent behavior identification result, which is used to supplement the participant's non-voice participation data in the subsequent construction of the meeting behavior credibility matrix.
[0065] In this embodiment S4, the meeting behavior credibility matrix refers to a two-dimensional matrix structure established with the participant's identity as the row index and the meeting time slice as the column index. The value of the meeting behavior credibility matrix is jointly calculated by the explicit speech attribution result and the implicit behavior identification result, representing the comprehensive credibility of the participant's speaking behavior and collaborative behavior within a specific time slice.
[0066] The method for constructing the meeting behavior credibility matrix is as follows: within each time slice, the path matching degree of the explicit speech attribution result and the non-voice contribution intensity value in the implicit behavior recognition result are weighted and fused to calculate the behavior credibility value of the participant in that time slice, and written into the matrix unit to form the meeting behavior credibility matrix.
[0067] In this embodiment S4, the temporal consistency verification refers to the process of logically checking the changes in the credibility of each participant's behavior in consecutive time slices after the meeting behavior credibility matrix is constructed, based on the participant's identity index and time slice order. This is used to verify the continuity and uniqueness of the same participant's behavior in the time series dimension. The system compares the trend of behavior credibility changes in adjacent time slices by traversing the row index of the matrix. If it finds that the same participant has multiple parallel mapping records of meeting decision content or abnormal mutations in behavior credibility in the same time slice, the consistency verification module is triggered to perform conflict detection and elimination. The execution of this temporal consistency verification ensures that only one credible identity mapping result is retained in each time slice of the matrix, avoiding identification biases such as duplicate speaking or false participation at the same time point.
[0068] In this embodiment S4, the specific method for verifying the temporal consistency of the meeting behavior credibility matrix based on the participant's identity is as follows: The system uses the participant's identity as the index unit and performs monotonicity and uniqueness checks on the behavior credibility sequence of that identity across all time slices. The monotonicity check is used to determine whether the continuous change of behavior credibility in the time series conforms to the actual speaking and collaboration rhythm in the meeting, and abnormal jumps are identified by detecting whether the mutation rate of behavior credibility between consecutive time slices exceeds a threshold. The uniqueness check is used to determine whether the identity appears simultaneously in multiple mapping records of meeting decision content in each time slice. If the same identity has two or more behavior credibility values in the same time slice, the system selects the highest value as the valid record according to the confidence weight and deletes the remaining records to maintain the consistency of the participants' credibility. Single behavior attribution of identity; multiple mappings of participant identities to meeting decision content within the same time slice. Duplicate mapping records refer to situations where multiple matrix units in the same column of the meeting behavior credibility matrix point to different meeting decision content nodes but have the same participant identity index. This indicates that the same identity was detected simultaneously participating in multiple decision items' speeches or collaborations within that time slice. Since meeting behaviors cannot be parallelized in the physical time dimension, a participant can only correspond to one valid behavior record within the same time slice. Therefore, such duplicate mapping records are considered logically conflicting data. When performing temporal consistency verification, the system removes these duplicate mapping records, retaining only the record with the highest behavior credibility value and deleting the remaining conflicting items to ensure that the behavior attribution reflected in the meeting behavior credibility matrix is unique and temporally interpretable.
[0069] Example 2: The intelligent voice conference behavior analysis system based on multi-source perception proposed in this invention is applied to the intelligent voice conference behavior analysis method based on multi-source perception proposed in Example 1. It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the intelligent voice conference behavior analysis method based on multi-source perception in Example 1.
[0070] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.
Claims
1. A method for intelligent voice conferencing behavior analysis based on multi-source perception, characterized in that, Includes the following steps: S1. Collect audio signals, collaborative behavior data and terminal device data during the meeting, and use the meeting timestamp dynamic window segmentation mechanism to construct a time slice index data structure, voice spatial positioning sequence and terminal activation record sequence; S2. Extract the sound source direction and intensity vector of the audio signal, construct the activation state correspondence matrix between the conference speakers and the device by combining terminal device data, construct a multi-source collaborative attribution graph structure based on the speech spatial localization sequence and the terminal activation record sequence, and use the multi-dimensional path similarity comparison method in the multi-source collaborative attribution graph structure to determine the optimal mapping relationship between the audio signal and the participants, and obtain the explicit speech attribution result. In S2, the multi-source collaborative attribution graph structure refers to a directed weighted graph structure with participant identity and time slice index as nodes and the response coupling relationship between audio signal and terminal device as edges, used to describe the attribution path relationship between audio signal and participant identity. S3. Identify non-voice input behaviors based on collaborative behavior data and construct a meeting collaboration graph structure. Perform semantic consistency comparison between non-voice input behaviors and time slice index data structure. Use the structural coupling degree function to calculate the structural contribution strength of non-voice input behaviors to the meeting decision content. Calculate the non-voice contribution strength sequence based on the structural contribution strength and the weight of each node in the meeting collaboration graph to obtain the implicit behavior identification result. In S3, the meeting collaboration graph structure refers to a directed weighted graph structure with participant identities and collaborative behavior events as nodes and collaborative relationships and event dependencies as edges, used to describe the operation and interaction paths between participants and between participants and meeting content in the meeting. S4. Construct a meeting behavior credibility matrix based on explicit speech attribution results and implicit behavior recognition results. Perform temporal consistency verification on the meeting behavior credibility matrix based on the participant identity. Eliminate duplicate mapping records of meeting decision content with multiple mappings of participant identity in the same time slice to obtain the meeting behavior credibility recognition results.
2. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 1, characterized in that: In S1, the meeting timestamp dynamic window segmentation mechanism refers to a control function that uses the meeting timeline as an index reference and calculates and dynamically determines the window duration based on the audio energy change rate and the number of collaborative behavior triggers. When the window duration reaches a preset time threshold and the number of collaborative behavior triggers reaches a cumulative threshold, the window duration is updated to generate a new time slice. This is used to segment and synchronize the audio signals, collaborative behavior data, and terminal device data in the meeting, and to construct a time slice index data structure, a voice spatial positioning sequence, and a terminal activation record sequence.
3. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 2, characterized in that: In S2, the activation state correspondence matrix refers to a two-dimensional matrix structure established with the time slice index as the row index and the terminal device and audio signal features as the column index. The values of the activation state correspondence matrix are the microphone activation state of the corresponding terminal device in the time slice, the level change, and the correlation strength between them and the peak value of the audio signal energy. The method for constructing the activation state correspondence matrix is as follows: read terminal device data, calculate the average audio input power and level change rate of each terminal device in the corresponding time slice, and perform normalization matching with the speech spatial positioning sequence to obtain the value of the activation state correspondence matrix, thereby constructing the activation state correspondence matrix.
4. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 3, characterized in that: The method for constructing the multi-source collaborative attribution graph structure is as follows: Based on the activation state correspondence matrix, the correlation strength between each terminal device and the direction of the sound source is extracted as the basis for edge weight calculation. The edge weight is defined as a linear combination of the sound source azimuth angle difference and the terminal device response delay. A connection edge is established between the identity of each participant and the time slice index within the corresponding time slice to form the multi-source collaborative attribution graph structure.
5. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 4, characterized in that: In S2, the multi-dimensional path similarity comparison method refers to calculating the path similarity from each participant's identity to the time slice index path in the multi-source collaborative attribution graph structure, using sound source direction features, sound intensity vector features, and terminal device activation delay features as comparison dimensions, and using a weighted cosine similarity function to obtain the path matching degree. The optimal mapping relationship refers to selecting the path with the highest path matching degree and edge weight consistency that meets the threshold constraint among all paths from participant identity to time slice index, as the unique mapping path between audio signal and participant identity, which is the optimal mapping relationship. The explicit speech attribution result refers to the paired records of the optimal mapping relationship, which are used to determine the explicit attribution result between the speaker's identity and the speech content during the meeting behavior analysis process.
6. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 5, characterized in that: The method for constructing the meeting collaboration graph structure is as follows: Based on the collaborative behavior event data, establish connection edges between participant identity nodes and corresponding collaborative behavior events, take the order of operation time as the direction attribute of the edge, and take the similarity of operation content and the normalized reciprocal of the time interval between operations as the edge weight; perform edge aggregation processing on event nodes that share the same operation goal among different participant identities to obtain the meeting collaboration graph structure.
7. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 6, characterized in that: In S3, the structural coupling function is a function that calculates the structural correlation strength between non-voice input behavior and meeting decision content based on the meeting collaboration graph structure, and is used to calculate the structural contribution strength of non-voice input behavior to meeting decision content. The calculation process of the structural coupling degree function is as follows: calculate the semantic similarity between the feature vectors of collaborative behavior event nodes and meeting decision content, calculate the path decay factor in combination with the shortest path length in the meeting collaboration graph structure, and use the weighted product of the path decay factor and semantic similarity as the structural coupling degree value of the collaborative behavior event node. By calculating the structural coupling degree values of all collaborative behavior event nodes, the structural contribution strength of non-voice input behavior to the meeting decision content is obtained.
8. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 7, characterized in that: In S3, the non-speech contribution intensity sequence refers to the structural contribution intensity sequence of non-speech input behavior in each time slice, recorded in the order of time slice index. The calculation method for the non-voice contribution intensity sequence is as follows: based on the structural contribution intensity value of each collaborative behavior event node, the weighted sum is calculated according to the edge weight between the participant identity node and the collaborative event node in the meeting collaboration graph, and the results are arranged in time slice order to form the non-voice contribution intensity sequence. The implicit behavior recognition result refers to determining, based on the time slice range in the non-voice contribution intensity sequence that continuously exceeds a preset intensity threshold, whether the corresponding participant has a collaborative behavior with high structural contribution intensity within that time slice, and marking the collaborative behavior as an implicit behavior recognition result.
9. The intelligent voice conferencing behavior analysis method based on multi-source perception according to claim 8, characterized in that: In S4, the meeting behavior credibility matrix refers to a two-dimensional matrix structure established with the participant's identity as the row index and the meeting time slice as the column index. The value of the meeting behavior credibility matrix is jointly calculated by the explicit speech attribution result and the implicit behavior identification result, representing the comprehensive credibility of the participant's speaking behavior and collaborative behavior within a specific time slice. The method for constructing the meeting behavior credibility matrix is as follows: within each time slice, the path matching degree of the explicit speech attribution result and the non-voice contribution intensity value in the implicit behavior recognition result are weighted and fused to calculate the behavior credibility value of the participant in that time slice, and written into the matrix unit to form the meeting behavior credibility matrix.
10. A multi-source perception-based intelligent voice conferencing behavior analysis system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes a computer program to implement the intelligent voice conference behavior analysis method based on multi-source perception as described in any one of claims 1-9.