Meeting validity identification method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment and storage medium for identifying the effectiveness of a meeting, the method comprising: extracting features from pre-collected audio information and video information, arranging the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence; matching and fusing the extracted audio features and video features to obtain audio and video feature information; calculating a speech effectiveness index and a listening effectiveness index based on the audio and video feature information; analyzing the effectiveness of a meeting based on the speech effectiveness index and the listening effectiveness index, and generating optimization suggestions for the meeting based on the analysis results. By introducing video feature information, the present application effectively makes up for the shortcomings of a single audio collection method and improves the accuracy and practicality of identifying the effectiveness of a meeting.

Description

Technical Field

[0001] The present application relates to the field of audio and video analysis technology, and in particular to a conference effectiveness identification method, device, equipment and storage medium. Background Art

[0002] In the current field of conference effectiveness recognition technology, most mainstream methods rely on audio collection results during the meeting to perform simple recognition. However, this method has significant limitations in the recognition process. Specifically, the effectiveness of the meeting is identified by judging the strength, frequency and other characteristics of the audio signal. However, this basis for judgment is relatively single and easily interfered by external noise. For example, when there are other sounds or background noise in the meeting environment, the audio collection results may be seriously affected, resulting in a significant reduction in the sensitivity and accuracy of the recognition results. In addition, the effectiveness of the meeting is not only reflected in the volume and frequency of the sound. Therefore, the current method of identifying the effectiveness of the meeting through audio collection results often cannot fully and accurately reflect the effectiveness of the actual meeting, and there are certain technical bottlenecks. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, equipment and storage medium for identifying the effectiveness of a meeting, aiming to solve the technical problem that the current method of identifying the effectiveness of a meeting through audio collection results cannot accurately reflect the actual effectiveness of the meeting.

[0004] To achieve the above object, the present application provides a method for identifying the validity of a meeting, the method comprising the following steps:

[0005] Extracting features from the pre-collected audio information and video information, and arranging the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence;

[0006] Match and fuse the extracted audio features and video features to obtain audio and video feature information;

[0007] Calculating a speaking effectiveness index and a listening effectiveness index according to the audio and video feature information;

[0008] The speaking effectiveness index and the listening effectiveness index are comprehensively used to perform a meeting effectiveness analysis, and optimization suggestions for the meeting are generated based on the analysis results.

[0009] In one embodiment, the step of extracting features from pre-collected video information and arranging the extracted video features in chronological order to form a video feature sequence includes:

[0010] Splitting the video information into video frames, performing pupil feature extraction, facial feature extraction and motion feature extraction on each video frame, and obtaining pupil features, facial features and motion features of each video frame;

[0011] The features extracted from each frame are arranged in chronological order to form a video feature sequence.

[0012] In one embodiment, pupil feature extraction, facial feature extraction and motion feature extraction are performed on each video frame, and the steps of obtaining pupil features, facial features and motion features of each video frame include:

[0013] Locating an eye region of a video frame, detecting pupils within the eye region, and calculating pupil features;

[0014] Locating a face region of a video frame, and calculating geometric features and texture features of the face based on feature points marked in the face region;

[0015] A human body posture estimation algorithm is used to detect key points of the human body in the video frame, and the motion features of the key points of the human body are obtained.

[0016] In one embodiment, the step of matching and fusing the extracted audio features and video features to obtain audio and video feature information includes:

[0017] Extracting the timestamps of the audio information and the video information, finding matching points by comparing the timestamps of the audio information and the video information, and performing preliminary alignment of audio features and video features based on the matching points;

[0018] Using a dynamic time warping algorithm, finely aligning the audio feature sequence and the video feature sequence;

[0019] The aligned audio features and video features are fused to obtain audio and video feature information.

[0020] In one embodiment, the step of calculating the speaking effectiveness index and the listening effectiveness index according to the audio and video feature information includes:

[0021] Analyzing the correlation between volume changes and facial expression changes in the audio and video feature information to identify speakers and listeners in the meeting;

[0022] The speaking effectiveness index includes speaking efficiency, speaking concentration and abnormal behavior rate, and the listening effectiveness index includes listening efficiency, listening concentration and abnormal behavior rate. The speaking efficiency, speaking concentration and abnormal behavior rate of the speaker and the listening efficiency, listening concentration and abnormal behavior rate of the listener are calculated respectively.

[0023] In one embodiment, the step of analyzing the correlation between volume changes and facial expression changes in the audio and video feature information to identify speakers and listeners in the meeting includes:

[0024] Identify participants with a significant increase in volume and a corresponding change in facial expression as speakers;

[0025] Participants with low or no noticeable volume changes and stable facial expressions were identified as listeners.

[0026] In one embodiment, the step of extracting features from the pre-collected audio information and video information, and arranging the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence, further includes:

[0027] The audio information and the video information in the conference are collected, and the audio information and the video information are pre-processed.

[0028] In addition, to achieve the above-mentioned purpose, the present application also provides a conference validity identification device, the conference validity identification device comprising:

[0029] The audio and video analysis module is used to extract features from the pre-collected audio information and video information, and arrange the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence;

[0030] The audio and video fusion module is used to match and fuse the extracted audio features and video features to obtain audio and video feature information;

[0031] An index calculation module, used to calculate a speaking effectiveness index and a listening effectiveness index according to the audio and video feature information;

[0032] The effectiveness analysis module is used to comprehensively analyze the effectiveness of the speech and the effectiveness of the listening, and generate optimization suggestions for the meeting based on the analysis results.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a terminal device, which includes a memory, a processor, and a meeting validity identification program stored in the memory and executable on the processor, and the meeting validity identification program, when executed by the processor, implements the steps of the meeting validity identification method as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a meeting validity identification program is stored. When the meeting validity identification program is executed by a processor, the steps of the meeting validity identification method as described above are implemented.

[0035] One or more technical solutions proposed in this application have at least the following technical effects:

[0036] This application adopts a technical solution for comprehensive analysis of audio and video. First, the pre-collected audio information and video information are respectively feature extracted. This step can capture the key information of the sound and picture in the meeting, and arrange them in chronological order to form a feature sequence. Further, the extracted audio features and video features are matched and fused to obtain richer audio and video feature information. This fusion not only enhances the comprehensiveness of the data, but also improves the accuracy and reliability of the feature information. On this basis, the speaking effectiveness index and the listening effectiveness index are calculated according to the audio and video feature information. These two indicators quantitatively evaluate the effectiveness of the meeting from the perspective of the speaker and the audience, respectively. Finally, the two indicators are combined to analyze the effectiveness of the meeting, and optimization suggestions for the meeting are generated based on the analysis results. This technical solution effectively makes up for the shortcomings of the single audio collection method by introducing video feature information, improves the accuracy and practicality of meeting effectiveness identification, and provides strong technical support for improving the quality of meetings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flowchart of a first exemplary embodiment of the conference validity identification method of the present application;

[0038] Figure 2 A system flow chart of a second exemplary embodiment for identifying the validity of a meeting of this application;

[0039] Figure 3 This is a schematic diagram of the module structure of the conference validity identification device according to an embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the conference validity identification method in the embodiment of the present application.

[0041] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] The main technical solutions of this application are: extracting features from pre-collected audio information and video information, arranging the extracted audio features and video features in chronological order to form audio feature sequences and video feature sequences; matching and fusing the extracted audio features and video features to obtain audio and video feature information; calculating the speaking effectiveness index and the listening effectiveness index based on the audio and video feature information; analyzing the effectiveness of the meeting by combining the speaking effectiveness index and the listening effectiveness index, and generating optimization suggestions for the meeting based on the analysis results.

[0044] Compared with the current method of identifying the effectiveness of a meeting through audio collection results, which has the technical problem of not being able to accurately reflect the actual effectiveness of the meeting, this application extracts features from the pre-collected audio information and video information at the same time, not only considering the basic properties of the audio signal such as intensity and frequency, but also introducing visual features such as facial expressions and gestures in the video, and arranging these extracted audio features and video features in chronological order to form a feature sequence. Furthermore, the audio and video features are matched and fused to obtain richer and multi-dimensional audio and video feature information.

[0045] Specifically, the following are the detailed steps of the first exemplary embodiment of the conference validity identification method of this application:

[0046] Reference Figure 1 , Figure 1 This is a flowchart of the first exemplary embodiment of the conference validity identification method of the present application. In this embodiment, the conference validity identification method includes steps S10 to S40:

[0047] Step S10, extracting features from the pre-collected audio information and video information, and arranging the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence;

[0048] First, feature extraction is performed on the pre-collected audio information, and the extracted audio features are arranged in chronological order to form an audio feature sequence;

[0049] For audio feature extraction, the speaker's volume, speaking speed, intonation and other features are extracted. The volume can be measured by calculating the root mean square (RMS) energy of the audio signal, which can be expressed as:

[0050]

[0051] Where x[n] is the sample value of the audio signal and N is the total number of samples. This formula calculates the average power of the audio signal over a period of time and can be used to represent the average volume over that period of time.

[0052] Speech rate can be determined by counting the number of syllables per unit time;

[0053] Intonation can be identified by analyzing the pitch changes of the audio signal. The pitch is usually related to the fundamental frequency (F0) of the audio signal and can be calculated by autocorrelation, cepstrum or other pitch estimation algorithms. If the fundamental frequency is obtained, its change over time can be analyzed to determine the intonation. The calculation of intonation can be performed by the following steps: first calculate the fundamental frequency of each audio frame, then calculate the change of the fundamental frequency between consecutive frames, and identify the intonation based on the change pattern of the fundamental frequency. The formula for intonation can be expressed as:

[0054] Intonation = F (fundamental frequency change)

[0055] Here, F is a function that determines the intonation based on the changing pattern of the fundamental frequency.

[0056] The above features can reflect the speaker's speaking efficiency, clarity and emotional state.

[0057] The extracted audio features need to be arranged in chronological order to form an audio feature sequence. This is used to identify the speaker and listener, and to evaluate the effectiveness of the meeting. To achieve this goal, timestamps can be used to mark the start and end time of each audio segment to ensure that the feature sequence is aligned with the time of the original audio data.

[0058] After feature extraction, an audio feature sequence containing volume, speaking speed, intonation, etc. is obtained. This sequence not only contains the speaker's voice features, but also the time information of the speech, providing important data for subsequent audio and video fusion and conference effectiveness analysis.

[0059] Suppose that in a company's strategic planning meeting, five executives participated in the discussion. The audio of the entire meeting was collected using multiple microphones and preprocessed, including denoising and normalization. In the feature extraction stage, the audio clips of each speaker were analyzed to extract their volume, speaking speed, and intonation features. If it is found that executive A speaks at a high volume, moderate speed, and steady intonation, this indicates that he may be confident and authoritative when conveying important information. Executive B, on the other hand, has a low volume, fast speed, and large changes in intonation, which may indicate that he is nervous or excited during the discussion. By arranging these features in chronological order, we form a detailed audio feature sequence, which provides basic data for subsequent meeting effectiveness analysis. By analyzing this data, we can evaluate the speaking efficiency and concentration of each executive, as well as their response to the content of the meeting, and thus make targeted meeting optimization suggestions.

[0060] Further, feature extraction is performed on the pre-collected video information, and the extracted video features are arranged in chronological order to form a video feature sequence;

[0061] In a feasible implementation, performing feature extraction on pre-collected video information and arranging the extracted video features in chronological order to form a video feature sequence may include steps S11 to S12:

[0062] Step S11, splitting the video information into video frames, performing pupil feature extraction, facial feature extraction and motion feature extraction on each video frame, and obtaining pupil features, facial features and motion features of each video frame;

[0063] In a feasible implementation, step S11 may include steps S111 to S113:

[0064] Step S111, locating the eye region of the video frame, detecting the pupil in the eye region, and calculating pupil features;

[0065] Specifically, in each video frame, the Haar feature classifier in the OpenCV library is used to identify the general outline of the face. After the face area is located, the eye detection algorithm is further used to accurately identify the position of the eyes. Under the deep learning framework, using Dlib's 68-point facial feature detector or other models, the precise position of the eyes can be identified, including the approximate area of ​​the corners of the eyes, eyelids, and pupils.

[0066] After locating the eye region, pupil detection technology can be used to further process the eye region image to identify the center position and size of the pupil, and then an edge detection algorithm (such as the Canny edge detector) can be used to identify the pupil boundary. The pupil boundary is then refined to determine the center position and radius of the pupil. After the position of the pupil is determined, the pupil area, shape, and position relative to the eye can be calculated. The pupil area can be obtained by calculating the total number of pixels in the pupil area, the shape can be described by fitting an ellipse, and the position can be determined by the position coordinates of the center of the pupil relative to the eye region. These features will be recorded in each video frame to form a pupil feature sequence.

[0067] It should be noted that by analyzing the pupil feature sequence, the attention and emotional state of the participants can be assessed. For example, pupil dilation may indicate that the participants are interested or excited about the topic currently being discussed, while pupil constriction may indicate that they are bored or uninterested.

[0068] Step S112, locating the face area of ​​the video frame, and calculating the geometric features and texture features of the face based on the feature points marked in the face area;

[0069] Specifically, the face detection algorithm is used to locate the face area, identify the face contour and key feature points, such as the corners of the eyes, the tip of the nose, and the corners of the mouth. After obtaining these feature points, the geometric features of the face are calculated, including the distance between the eyes, the width of the nose, the size of the mouth, etc. These geometric features can be obtained by measuring the distance between the feature points. Facial texture features, such as skin texture and color changes, can be extracted by analyzing the changes in pixel intensity. These features can be extracted using methods such as local binary pattern (LBP) or Gabor filter.

[0070] By analyzing geometric and texture features, we can assess the emotions and engagement of attendees. For example, we can assess whether attendees are smiling by analyzing the degree of upturned corners of their mouths, and thus infer their emotional state. By analyzing the degree of raised eyebrows and open eyes, we can assess the degree of surprise or attention of attendees.

[0071] Step S113, using a human posture estimation algorithm to detect key points of the human body in the video frame, and obtaining motion features of the key points of the human body.

[0072] Specifically, human pose estimation algorithms are used to detect key points of the human body in video frames and obtain the motion features of these key points. Human pose estimation usually involves identifying various parts of the human body, such as the head, shoulders, elbows, wrists, waist, knees, etc., and tracking the movement of these parts over time. This can be achieved through traditional computer vision techniques such as background subtraction, optical flow, or deep learning-based methods such as pose estimation networks. The motion features of key points include changes in joint angles, velocity and acceleration of body parts, etc. These features can be obtained by calculating the relative positions and motion vectors between key points.

[0073] By analyzing the motion features at key points, it is possible to assess the behavior and engagement of participants. For example, frequent gestures may indicate that participants are actively speaking or expressing their opinions, while crossed arms may indicate that they have reservations or are uncomfortable with a topic.

[0074] Step S12, arranging the features extracted from each frame in chronological order to form a video feature sequence.

[0075] Specifically, we first need to ensure that all extracted features correspond to the timestamps in the video frames. Each feature point needs to be marked with the video frame number to which it belongs and the specific time point in the frame. This is done to maintain the temporal continuity of the features and ensure that they can be accurately mapped to specific meeting moments in subsequent analysis.

[0076] Next, the pupil features extracted from each frame are sorted, including the pupil area, shape, and position. This data will be recorded as a sequence, reflecting the changes in the pupil state of each participant during the meeting. Similarly, facial features, such as the distance between the eyes, the width of the nose, the size of the mouth, and the texture and color changes of the skin, will also be sorted into sequences. This data will reveal the changes in the facial expressions and emotions of the participants.

[0077] Action features, including changes in joint angles, speed and acceleration of body parts, also need to be integrated into the video feature sequence. This involves tracking key points of the human body and calculating motion vectors to obtain dynamic information about the participant's body language and posture.

[0078] When integrating these features, we need to pay attention to the synchronization of data. Since feature extraction may be completed at different time points, we need to ensure that all feature sequences are aligned in time. This may involve interpolation or smoothing of some feature sequences to ensure the continuity and consistency of the data.

[0079] Finally, these feature sequences are saved in a structured data format, such as a CSV file or a database, for further analysis. This video feature sequence will provide basic data for the audio and video fusion module, enabling it to combine audio features with video features for a more in-depth analysis of the effectiveness of the meeting.

[0080] Step S20, matching and fusing the extracted audio features and video features to obtain audio and video feature information;

[0081] In a feasible implementation, step S20 may include steps S21 to S23:

[0082] Step S21, extracting the timestamps of the audio information and the video information, finding matching points by comparing the timestamps of the audio information and the video information, and performing preliminary alignment of audio features and video features based on the matching points;

[0083] Specifically, we first need to extract timestamp information from the audio and video feature sequences. These timestamps mark the exact location of each feature point in the original audio and video streams. For audio features, the timestamp may indicate the moment when a specific volume or tone change occurs; for video features, the timestamp may indicate the moment when a specific facial expression or action occurs.

[0084] Next, matching points are found by comparing these timestamps. This process involves scanning the audio and video feature sequences for feature points that are close or coincident in time. For example, if at a certain point in time, the audio feature shows a significant increase in volume, and the video feature shows a corresponding change in facial expression, then the two feature points can be considered a match.

[0085] After finding the matching points, the audio features and video features are initially aligned so that both information can be considered simultaneously in subsequent analysis. This alignment can be based on the nearest neighbor principle, that is, selecting the feature points closest in time for matching; or it can be based on more complex pattern recognition algorithms to identify specific audio and video events.

[0086] After the initial alignment, a paired list of audio and video features can be obtained, which will serve as the basis for the next step of fine alignment.

[0087] Assume that in a company's year-end summary meeting, the audio and video data of the meeting are recorded, and the audio and video features are extracted. In step S21, the timestamp information of volume, speech speed and tone is extracted from the audio feature sequence, and the timestamp information of pupil changes, facial expressions and body movements is also extracted from the video feature sequence.

[0088] When comparing these timestamps, it was found that at a specific moment in the meeting, the audio features showed that the volume of a speaker suddenly increased, while the video features showed that the speaker also made obvious gestures at the same time. By matching these feature points, the audio and video features were initially aligned, and the increase in volume was associated with gestures, which may indicate that the speaker was emphasizing an important point.

[0089] Step S22, using a dynamic time warping algorithm to finely align the audio feature sequence and the video feature sequence;

[0090] Dynamic Time Warping (DTW) is an algorithm used to measure the similarity between two time series, especially when the two sequences may be stretched or deformed in time. In the alignment of audio and video feature sequences, the DTW algorithm can find the best match.

[0091] The dynamic time warping algorithm is based on the observation that two similar time series may not be completely aligned in time, but by "stretching" or "compressing" them on the time axis, they can be made more similar in shape. The DTW algorithm achieves alignment by calculating the distance between the two series and trying to find a path that minimizes the cumulative distance on this path.

[0092] Specifically, we first initialize and create a DTW distance matrix whose size is the product of the lengths of the two feature sequences. The rows of the matrix represent the time points of one sequence, and the columns represent the time points of the other sequence.

[0093] Fill the matrix. The value of each cell (i, j) of the matrix is ​​calculated in the following way. It represents the distance between the i-th and j-th features of the two sequences. The calculation formula of (i, j) is as follows:

[0094] D(i,j)=D(i-1,j-1)+d(X(i),Y(j))

[0095] Where D is the DTW distance matrix, d is the distance metric between features (such as Euclidean distance), and X(i) and Y(j) are the i-th and j-th features of the two sequences, respectively.

[0096] Find the optimal path, starting from the upper left corner of the matrix, through the matrix to the lower right corner, and find a path that minimizes the cumulative distance on the path. This path represents the best alignment between the two sequences.

[0097] The path is traced back, starting from the lower right corner of the matrix, and following the path with the smallest cumulative distance until the upper left corner. This path indicates the best way to match the features in the two sequences.

[0098] Aligning feature sequences,According to the optimal path, the audio and video feature sequences are realigned,so that they correspond in time.

[0099] After alignment, a temporally accurate audio and video feature sequence is obtained, which will provide more accurate data support for subsequent conference effectiveness analysis because it takes into account the actual temporal relationship between audio and video data.

[0100] For example, it was noticed that at a certain moment in the meeting, the audio features of a speaker showed a sudden increase in volume, but no corresponding facial expression change was observed in the initially aligned video features. By applying the DTW algorithm, the best matching path was found, and it was discovered that there was actually a brief facial expression change in the video features corresponding to the increase in volume, but due to the difference in video frame rate and audio sampling rate, this change was ignored in the initial alignment. The DTW algorithm adjusts the timeline to accurately align this key facial expression change with the audio features, thereby obtaining a more accurate audio and video feature sequence.

[0101] Step S23, fusing the aligned audio features and video features to obtain audio and video feature information.

[0102] It should be noted that during the fusion process, you may need to deal with the scale and unit issues of different types of features. For example, volume is a continuous value, while facial expressions may be categorical variables. Therefore, these features may need to be standardized or normalized so that they can be used in the same model. In addition, feature selection and dimensionality reduction may also need to be considered to reduce redundancy and improve the efficiency of analysis.

[0103] The fused audio and video feature information will provide a comprehensive view of the communication dynamics at every moment in the meeting. This information can be used to evaluate the effectiveness of the meeting, identify key communication moments, and make recommendations to improve meeting efficiency.

[0104] Step S30, calculating a speaking effectiveness index and a listening effectiveness index according to the audio and video feature information;

[0105] In a feasible implementation, step S30 may include steps S31 to S32:

[0106] Step S31, analyzing the correlation between volume changes and facial expression changes in the audio and video feature information to identify speakers and listeners in the meeting;

[0107] In a feasible implementation, step S31 may include steps S311 to S312:

[0108] Step S311, identifying the participant whose volume increases significantly and whose facial expression changes accordingly as the speaker;

[0109] Specifically, first identify the moment when the volume increases significantly from the audio feature sequence. This can be achieved by calculating the root mean square (RMS) energy or peak volume of the audio signal and comparing it with a preset threshold to determine whether the volume has increased significantly. At the same time, analyze the changes in facial expressions from the video feature sequence, which usually involves comparing the changes in key facial feature points in consecutive video frames, such as the rise of the corners of the mouth, the degree of eye openness, etc., to identify changes in expression.

[0110] To correlate volume changes with facial expression changes, check whether the moment of volume increase coincides with a specific facial expression change. This association can be achieved using timestamp-aligned audio and video feature vectors. If a participant shows a facial expression change associated with speaking (e.g., mouth opening, eyes looking forward, etc.) at the same time as the volume increase, the participant is identified as the speaker.

[0111] Step S312: Identify the participant whose volume is low or has no obvious change and whose facial expression is stable as a listener.

[0112] This step focuses on participants with low or no significant volume changes and stable facial expressions. First, the periods of low or stable volume are selected from the audio feature sequence and compared with the frames in the video feature sequence that show stable facial expressions. The stability of facial expressions can be evaluated by calculating the magnitude of the change in facial feature points between consecutive frames. If the magnitude of the change is below a certain threshold, the facial expression can be considered stable.

[0113] In this step, look for participants who show low activity levels in both volume and facial expressions. For example, if a participant's audio features show low and stable volume, while the video features show no significant changes in facial expression (such as no obvious mouth opening or large eye movements), then the participant is identified as a listener. This process helps distinguish between active participants and passive listeners in the meeting, thereby more accurately assessing the interactivity and engagement of the meeting.

[0114] Step S32, the speaking effectiveness indicators include speaking efficiency, speaking concentration and abnormal behavior rate, and the listening effectiveness indicators include listening efficiency, listening concentration and abnormal behavior rate. The speaker's speaking efficiency, speaking concentration and abnormal behavior rate and the listener's listening efficiency, listening concentration and abnormal behavior rate are calculated respectively.

[0115] Speech efficiency is calculated by analyzing the speaker's volume. First, the total speaking time in the meeting is determined, and then the speaker's volume is calculated to exceed a certain threshold. For example, if the speaker's volume is not less than 80% of the average volume, it is considered to be effective. The formula for calculating speech efficiency can be expressed as:

[0116]

[0117] Among them, the higher this ratio is, the more efficient the speaker is in the meeting.

[0118] In addition, whether the speech is valid can also be determined by judging whether the voice is loud. The judgment of the loudness of the voice can be made by comparing the loud tones of the respective gender characteristics.

[0119] The speech concentration is calculated by analyzing the speaker's facial expression and pupil state. The proportion of time that the speaker's facial expression matches the concentration expression library and the proportion of time that the pupil is in a focused state during the speech are determined. The formula can be expressed as:

[0120]

[0121] If this ratio exceeds 70%, the speaker is considered to have a high level of concentration.

[0122] The abnormal behavior rate is calculated by analyzing the speaker's hand movements, determining the number of reasonable hand movements of the speaker, and then calculating the proportion of abnormal movements. The formula can be expressed as:

[0123]

[0124] The lower this ratio is, the more natural the speaker's overall behavior is.

[0125] Listening efficiency is calculated by analyzing the volume of the listener during non-speaking periods. The proportion of time when the listener's volume is below a certain threshold when the speaker is speaking is determined. The formula can be expressed as:

[0126]

[0127] The higher this ratio is, the more efficient the listener is.

[0128] Listening concentration is calculated by analyzing the listener's facial expression and pupil state. Facial micro-expression recognition technology can be used to compare the similarity of the listener's facial expression with the smiley face library to evaluate whether his expression is serious, and then determine the proportion of time that the listener's facial expression matches the serious expression library, and the proportion of time that the pupil is in a focused state during the speech. The formula can be expressed as:

[0129]

[0130] If this ratio exceeds 80%, the listener's concentration is considered to be high.

[0131] The abnormal listening behavior rate is calculated by analyzing the body movements of the listener. The number of reasonable body movements of the listener is determined, and then the proportion of abnormal movements is calculated. Image recognition technology can be used to monitor the body movements of the listener, especially whether there are large swings, such as whispering. The formula can be expressed as:

[0132]

[0133] The lower this ratio, the more natural the listener's overall behavior.

[0134] Step S40, comprehensively analyzing the effectiveness of the speech and the effectiveness of the listening, and generating optimization suggestions for the meeting based on the analysis results.

[0135] Specifically, for the quality of the meeting, after calculating the speaking efficiency and listening efficiency, a ratio of 6:4 is used as the calculation standard for the overall quality of the meeting. Speaking efficiency reflects the speaker's activity and participation in the meeting, while listening efficiency reflects the listener's attention to the content of the meeting. Define the following formula to calculate the overall quality of the meeting:

[0136] Meeting quality = 0.6 × speaking efficiency + 0.4 × listening efficiency

[0137] Then, calculate the meeting concentration. This indicator first calculates the speaking concentration and listening concentration, and uses a 4:6 ratio of unit time length as the calculation standard. The speaking concentration and listening concentration are calculated based on the facial expressions, pupil focus and other features of the speaker and listener. Define the following formula to calculate the overall meeting concentration:

[0138] Meeting concentration = 0.4 × speaking concentration + 0.6 × listening concentration

[0139] Then, calculate the abnormal behavior rate of the meeting. This indicator first calculates the abnormal behavior rate of speech and the abnormal behavior rate, and takes their median average as the evaluation criterion. The abnormal behavior rate is calculated based on the hand movements, body movements and other characteristics of the speaker and listener. Define the following formula to calculate the abnormal behavior statistics of the meeting:

[0140]

[0141] Finally, calculate the noise interference rate in the meeting. This indicator takes the percentage of video image and audio noise removed in the audio and video fusion process as the calculation standard. The following formula can be defined to calculate the noise and interference rate in the meeting:

[0142]

[0143] For example, in a company's quarterly performance evaluation meeting, by analyzing the audio and video data, we found that the speaking efficiency was 0.8 and the listening efficiency was 0.6. According to the formula, we calculated the meeting quality as:

[0144] Meeting quality = 0.6 × 0.8 + 0.4 × 0.6 = 0.72

[0145] Similarly, it is found that the speaking concentration is 0.75 and the listening concentration is 0.5. The meeting concentration is calculated as:

[0146] Meeting concentration = 0.4 × 0.75 + 0.6 × 0.5 = 0.6

[0147] The abnormal behavior rate for speaking is 0.1, the abnormal behavior rate for listening is 0.2, and the abnormal behavior rate for meetings is:

[0148] Abnormal behavior rate in meetings = (0.1 + 0.2) / 2 = 0.15

[0149] During the audio and video fusion process, 15% of the noise was removed, so the noise and interference rate in the meeting was 15%.

[0150] Based on these analysis results, the following meeting optimization suggestions were generated: improving meeting speaking efficiency, which may be achieved by encouraging more interaction and discussion; increasing listening concentration, which may be achieved by providing more attractive presentation content; reducing abnormal behavior, which may be achieved by clarifying meeting rules and expected behavior; reducing noise and interference, which may be achieved by improving the meeting environment and equipment. These suggestions are aimed at improving the overall quality and efficiency of the meeting and ensuring the smooth realization of the meeting goals.

[0151] Further, based on Figure 1 In the second exemplary embodiment of the present application, step S10: extracting features from pre-collected audio information and video information, arranging the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence, further includes step S50:

[0152] Step S50: collecting audio information and video information in the conference, and pre-processing the audio information and the video information.

[0153] Specifically, first deploy audio and video acquisition equipment. For audio information, multiple high-sensitivity microphones can be used to capture speeches in the meeting. These microphones are evenly distributed in the meeting room to ensure that sounds from different locations can be clearly recorded. For video information, a high-definition camera can be used to record the meeting process. The camera can cover the entire meeting room and capture the expressions and movements of all participants. These devices should be installed and tested in advance to ensure that they can operate stably during the meeting.

[0154] The collected raw audio and video data usually contains noise and unnecessary information, so it needs to be preprocessed. The first step in audio preprocessing is noise reduction, which can be done through spectral subtraction, Wiener filtering, or deep learning noise reduction techniques. The denoised audio signal will be clearer, which helps with subsequent feature extraction. Next, the audio signal is normalized to ensure consistent volume levels, which helps to make fair comparisons between different speakers.

[0155] Video preprocessing includes frame rate adjustment, resolution scaling and compression. Frame rate adjustment ensures smooth video playback, while resolution scaling and compression help reduce the amount of data for subsequent processing and storage. In addition, video data can be color corrected and brightness adjusted to improve image quality and make facial expressions and movement features easier to recognize.

[0156] After the audio and video data are preprocessed, since the audio and video data may be collected by different devices, their timestamps may not be completely consistent. Therefore, an external clock signal can be used or timestamps can be embedded in the data to ensure that the audio and video data are aligned in time. This is crucial for subsequent audio and video feature matching and fusion.

[0157] In addition, the present application also proposes a conference validity identification device, the conference validity identification device comprising:

[0158] The audio and video analysis module 10 is used to extract features from the pre-collected audio information and video information, and arrange the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence;

[0159] The audio and video fusion module 20 is used to match and fuse the extracted audio features and video features to obtain audio and video feature information;

[0160] An index calculation module 30, used to calculate a speaking effectiveness index and a listening effectiveness index according to the audio and video feature information;

[0161] The effectiveness analysis module 40 is used to comprehensively analyze the effectiveness of the speech and the effectiveness of the listening, and generate optimization suggestions for the meeting based on the analysis results.

[0162] The conference validity identification device provided by the present application adopts the conference validity identification method in the above-mentioned embodiment, aiming to solve the technical problem that the current method of conference validity identification through audio acquisition results cannot accurately reflect the actual validity of the meeting. Compared with the prior art, the beneficial effects of the conference validity identification device provided by the present application are the same as the beneficial effects of the conference validity identification method provided by the above-mentioned embodiment, and the other technical features in the conference validity identification device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0163] The present application provides a conference validity identification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the conference validity identification method in the above-mentioned embodiment one.

[0164] The conference validity identification device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 4The conference validity identification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0165] like Figure 4 As shown, the conference validity recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the conference validity recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the conference validity identification device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a conference validity identification device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0166] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0167] The conference validity identification device provided by the present application adopts the conference validity identification method in the above-mentioned embodiment, aiming to solve the technical problem that the current method of conference validity identification through audio acquisition results cannot accurately reflect the actual validity of the meeting. Compared with the prior art, the beneficial effects of the conference validity identification device provided by the present application are the same as the beneficial effects of the conference validity identification method provided by the above-mentioned embodiment, and the other technical features in the conference validity identification device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0168] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0169] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0170] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the conference validity identification method in the above-mentioned embodiment.

[0171] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0172] The computer-readable storage medium may be included in the conference validity identification device; or may exist independently without being assembled into the conference validity identification device.

[0173] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0176] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned conference validity identification method, and is intended to solve the technical problem that the current method of conference validity identification through audio collection results cannot accurately reflect the actual validity of the meeting. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the conference validity identification method provided in the above-mentioned embodiment, and will not be repeated here.

[0177] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned conference effectiveness identification method when executed by a processor.

[0178] The computer program product provided by this application is intended to solve the technical problem that the current method of identifying the effectiveness of a meeting through audio collection results cannot accurately reflect the actual effectiveness of the meeting. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the method for identifying the effectiveness of a meeting provided by the above-mentioned embodiment, and will not be elaborated here.

[0179] Compared with the prior art, the conference validity identification method, device, equipment, medium and computer product proposed in the embodiment of the present application extracts the business characteristic information of the target business, performs data standardization processing on the business characteristic information to obtain standard characteristic data, performs hash processing on the standard characteristic data to obtain unique characteristic data, performs numerical processing and splicing processing on the unique characteristic data to obtain the first business characteristic value, accumulates the first business characteristic value of the target business to obtain the target business characteristic value, and finally compares the target business characteristic value with the characteristic value set to obtain the conference validity identification result. Compared with the traditional method of identifying repeated businesses by generating a unique key value or a continuous serial number for each business, it is more efficient, flexible and reliable. Based on the solution of the present application, the business of complex scenarios is transformed into a comparison of two numbers through a series of simple transformations, so that the comparison process is very intuitive and efficient. The system only needs to simply compare whether the two values ​​are equal to quickly determine whether the two businesses are exactly the same.

[0180] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0181] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0183] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for identifying conference effectiveness, characterized in that: The conference validity identification method comprises: Extracting features from the pre-collected audio information and video information, and arranging the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence; Match and fuse the extracted audio features and video features to obtain audio and video feature information; Calculating a speaking effectiveness index and a listening effectiveness index according to the audio and video feature information; The speaking effectiveness index and the listening effectiveness index are comprehensively used to perform a meeting effectiveness analysis, and optimization suggestions for the meeting are generated based on the analysis results.

2. The conference validity identification method according to claim 1, characterized in that: The steps of extracting features from pre-collected video information and arranging the extracted video features in chronological order to form a video feature sequence include: Splitting the video information into video frames, performing pupil feature extraction, facial feature extraction and motion feature extraction on each video frame, and obtaining pupil features, facial features and motion features of each video frame; The features extracted from each frame are arranged in chronological order to form a video feature sequence.

3. The method for identifying conference effectiveness as claimed in claim 2, characterized in that: Pupil feature extraction, facial feature extraction and motion feature extraction are performed on each video frame. The steps of obtaining pupil features, facial features and motion features of each video frame include: Locating an eye region of a video frame, detecting pupils within the eye region, and calculating pupil features; Locating a face region of a video frame, and calculating geometric features and texture features of the face based on feature points marked in the face region; A human body posture estimation algorithm is used to detect key points of the human body in the video frame, and the motion features of the key points of the human body are obtained.

4. The conference validity identification method according to claim 1, characterized in that: The step of matching and fusing the extracted audio features and video features to obtain audio and video feature information includes: Extracting the timestamps of the audio information and the video information, finding matching points by comparing the timestamps of the audio information and the video information, and performing preliminary alignment of audio features and video features based on the matching points; Using a dynamic time warping algorithm, finely aligning the audio feature sequence and the video feature sequence; The aligned audio features and video features are fused to obtain audio and video feature information.

5. The conference validity identification method according to claim 1, characterized in that: The step of calculating the speaking effectiveness index and the listening effectiveness index according to the audio and video feature information comprises: Analyzing the correlation between volume changes and facial expression changes in the audio and video feature information to identify speakers and listeners in the meeting; The speaking effectiveness index includes speaking efficiency, speaking concentration and abnormal behavior rate, and the listening effectiveness index includes listening efficiency, listening concentration and abnormal behavior rate. The speaking efficiency, speaking concentration and abnormal behavior rate of the speaker and the listening efficiency, listening concentration and abnormal behavior rate of the listener are calculated respectively.

6. The method for identifying conference effectiveness as claimed in claim 5, characterized in that: The step of analyzing the correlation between volume changes and facial expression changes in the audio and video feature information to identify speakers and listeners in the meeting includes: Identify participants with a significant increase in volume and a corresponding change in facial expression as speakers; Participants with low or no noticeable volume changes and stable facial expressions were identified as listeners.

7. The conference validity identification method according to claim 1, characterized in that: The step of extracting features from the pre-collected audio information and video information and arranging the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence also includes: The audio information and the video information in the conference are collected, and the audio information and the video information are pre-processed.

8. A conference validity identification device, characterized in that: The device comprises: The audio and video analysis module is used to extract features from the pre-collected audio information and video information, and arrange the extracted audio features and video features in chronological order to form an audio feature sequence and a video feature sequence; The audio and video fusion module is used to match and fuse the extracted audio features and video features to obtain audio and video feature information; An index calculation module, used to calculate a speaking effectiveness index and a listening effectiveness index according to the audio and video feature information; The effectiveness analysis module is used to comprehensively analyze the effectiveness of the speech and the effectiveness of the listening, and generate optimization suggestions for the meeting based on the analysis results.

9. A conference validity recognition device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the conference validity identification method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the conference validity identification method according to any one of claims 1 to 7 are implemented.

Images