A method and system for managing information of participants

Abstract

This application relates to the field of information management technology, and in particular to a method and system for managing the information of participants in a meeting. The method includes: collecting face images of multiple preset expressions of the participants in the meeting and audio information of preset statements; obtaining the voiceprint features of each participant based on the audio information; obtaining the face features of each preset expression, initializing the combination scheme of the preset expressions of each participant, and calculating the global salience and the individual salience of each participant; using the sum of the minimum values of the global salience and the individual salience as the objective function, and taking the combination scheme with the maximum value of the objective function as the target scheme; storing the voiceprint features of each participant, the target expressions in the target scheme, and the average face features of each target expression. Through the technical solution of this application, it is possible to screen the information of the participants in the meeting, reduce the information storage pressure while ensuring the accuracy of identity authentication, and realize the effective management of the information of the participants in the meeting.

Description

Technical Field

[0001] This application relates to the field of information management technology, and particularly to a method and system for managing information of participants in a meeting. Background Art

[0002] With the rapid development of Internet technology, online meetings have gradually replaced offline meetings due to their convenience and speed. During an online meeting, participants need to click on the meeting entry link on the client side and undergo identity authentication. Only when the identity authentication is successful can they enter the online meeting to ensure the security of the online meeting. To implement the identity authentication of participants, it is necessary to pre-enter the information of participants, and how to manage the information of participants to ensure the accuracy of identity authentication when entering the online meeting is an urgent problem to be solved.

[0003] Currently, the patent application document with the publication number CN113420274A discloses a system and method for access management of meeting participants based on trusted identity authentication. The method includes: identifying the document information and performing matching verification with the document registration information in the trusted identity authentication platform; prompting the meeting participants to perform an electronic signature and face verification, and collecting the posture and voice information of the meeting participants for comprehensive judgment; reading the third-party voice and retina biometric verification information according to the document information and storing it in the secure cache memory, and at the same time reading the voice and retina data of the meeting participants for information matching verification, and achieving trusted identity authentication through comprehensive local face matching and full-face matching.

[0004] The above method achieves trusted identity authentication by comprehensively considering various information such as the posture, voice information, retina data, and face information of the meeting participants. However, not all data is effective for identity authentication. Comprehensively considering multiple types of information will not only not increase the accuracy of identity authentication, but will instead increase the storage pressure and fail to effectively manage the information of the meeting participants. Summary of the Invention

[0005] To solve the technical problem of being unable to effectively manage the information of meeting participants, this application provides a method and system for managing the information of meeting participants, which can screen the information of meeting participants, reduce the information storage pressure while ensuring the accuracy of identity authentication, and achieve the effective management of the information of meeting participants.

[0006] In the first aspect of the present application, a method for managing information of participants is provided. The management method includes: collecting face images of multiple preset expressions of any participant and audio information of preset statements; dividing the audio information into audio segments of each character in the preset statement, obtaining the audio features of each character by using a long short-term memory network, and weighted summing the audio features of each character according to the variance of the audio features of each participant to obtain the voiceprint features of the participant; extracting features from the face images to obtain the face features of each preset expression, initializing the combination scheme of the preset expressions of each participant, calculating the global significance of the combination scheme and the individual significance of each participant according to the average face features of the selected preset expressions in the combination scheme, where the individual significance is the minimum difference in the average face features between the participant and other participants, and the global significance is negatively correlated with the variance of the individual significance; taking the sum of the global significance and the minimum value of the individual significance as the objective function, updating the combination scheme of each participant, and taking the combination scheme corresponding to the maximum value of the objective function as the target scheme; storing the voiceprint features of each participant, the target expressions in the target scheme, and the average face features of each target expression to implement the management of participant information, and the average face features and voiceprint features of each target expression are used for identity authentication.

[0007] Collect face images of multiple preset expressions of any participant and audio information of preset statements. The preset statement includes multiple characters. Use a long short-term memory network to extract the audio features of each character in the audio information. Since different participants may have different pronunciations when reading the same character, when all participants read a character, the greater the difference between the participants, the more accurately the audio features corresponding to the character can reflect the voiceprint features of each participant. Therefore, weighted sum the audio features of each character according to the variance of the audio features of each participant to obtain the voiceprint features of each participant; further, obtain the face features of each preset expression of a participant, initialize the combination scheme of the preset expressions of each participant. Each participant corresponds to a combination scheme. Calculate the global significance of the combination scheme and the individual significance of each participant. The individual significance can measure the difficulty of face recognition of the corresponding participant, and the global significance can measure the difference degree between the individual significances of all participants. Take the sum of the global significance and the minimum value of the individual significance as the objective function, and continuously update the combination scheme of each participant until the objective function reaches the maximum value to obtain the target scheme of each participant. According to the face features corresponding to the target expressions in the target scheme, the identity authentication of the corresponding participant can be accurately realized; store the voiceprint features of each participant, the target expressions in the target scheme, and the average face features of each target expression. Identity authentication based on the voiceprint features and the average face features of each target expression can reduce the storage pressure while ensuring the accuracy of identity authentication, and effectively manage the information of participants.

[0008] Preferably, dividing the audio information into audio segments of each character in the preset statement includes: performing frame division on the audio information and calculating the short-time energy at any timestamp to obtain a short-time energy sequence, dividing the short-time energy sequence based on the extreme points to obtain multiple time intervals; in response to the number of time intervals being equal to the number of characters in the preset statement, taking the audio information within each time interval as the audio segment of the corresponding character, otherwise, re-collecting the audio information of the participant.

[0009] When the number of time intervals is equal to the number of characters in the preset statement, a one-to-one correspondence between the time intervals and the characters is achieved; when the number of time intervals is not equal to the number of characters in the preset statement, it indicates that there are missing characters or repeated readings in the audio information. At this time, it is necessary to re-collect the audio information of the participant to ensure that the audio segment of each character of the participant can be obtained.

[0010] Preferably, obtaining the audio features of each character by using a long short-term memory network includes: inputting the audio information into the long short-term memory network to obtain short-term vectors at each timestamp; taking the average value of the short-term vectors at all timestamps within the audio segment of any character as the audio feature of the character.

[0011] Preferably, the participant 's voiceprint feature is:

[0012] , is the audio feature of the participant at the character , is the variance of the audio features of all participants' characters , is the sum of the variances of the audio features of all participants' all characters, is the number of characters in the preset statement.

[0013] When different participants read the same character, there will be different pronunciations. When all participants read a character, the greater the difference between the participants, the more accurately the audio feature corresponding to the character can reflect the voiceprint features of each participant. Therefore, the audio features of each character are weighted and summed according to the variance of the audio features of each participant to accurately obtain the voiceprint feature of each participant.

[0014] Preferably, the combination scheme is a multi-dimensional 01 vector. In response to the value of any dimension being 1, it means selecting the preset expression corresponding to that dimension, otherwise, not selecting the preset expression corresponding to that dimension.

[0015] Preferably, extracting features from the face image includes: extracting features from the face image according to the autoencoder network or the principal component analysis algorithm.

[0016] Preferably, for the calculation process of the individual saliency of the participants is as follows: Calculate the Euclidean distance between the average face features of a participant and those of other participants, and take the minimum Euclidean distance as the individual saliency of the participant .

[0017] The minimum value of the individual saliency can reflect the highest difficulty of the identity authentication of all participants. The larger the minimum value of the individual saliency, the smaller the highest difficulty, which means that the difficulty for all participants to complete the identity authentication is smaller, realizing the accurate quantification of the difficulty level of identity authentication.

[0018] Preferably, the global saliency satisfies the relational expression: , where is the variance of the individual saliency.

[0019] Preferably, the process of the identity authentication includes: in response to any participant receiving an invitation to join the meeting, sending an action instruction to the participant, where the action instruction includes the target expression of the participant; collecting the real-time voiceprint features and the real-time face features of each target expression, and calculating the real-time average face features; calculating the face similarity between the real-time average face features and the average face features, and the voiceprint similarity between the real-time voiceprint features and the voiceprint features. In response to both the face similarity and the voiceprint similarity being greater than a preset similarity, the identity authentication is successful; otherwise, the identity authentication fails.

[0020] Since the target solutions of each participant are different, the action instructions of each participant are also different. During the process of identity authentication, it is only necessary to collect the face images under the target expression, avoiding the collection of face images under all preset expressions, improving the authentication efficiency and reducing the calculation amount during identity authentication.

[0021] In the second aspect of the present application, a participant information management system is further provided, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method for managing participant information according to the first aspect of the present application is implemented.

[0022] The technical solution of the present application has the following beneficial technical effects:

[0023] First, collect the face images of multiple preset expressions of any participant and the audio information of preset statements. The preset statements include multiple characters. Use a long short-term memory network to extract the audio features of each character in the audio information. Since different participants may have different pronunciations when reading the same character, when all participants read a character, the greater the difference among the participants, the more accurately the audio features corresponding to the character can reflect the voiceprint features of each participant. Therefore, according to the variance of the audio features of each participant, the audio features of each character are weighted and summed to obtain the voiceprint features of each participant. Further, obtain the face features of each preset expression of a participant, initialize the combination scheme of the preset expressions of each participant. Each participant corresponds to a combination scheme. Calculate the global significance of the combination scheme and the individual significance of each participant. The individual significance can measure the difficulty of face recognition of the corresponding participant, and the global significance can measure the difference degree among the individual significances of all participants. Take the sum of the global significance and the minimum value of the individual significance as the objective function, and continuously update the combination scheme of each participant until the objective function reaches the maximum value to obtain the target scheme of each participant. According to the face features corresponding to the target expression in the target scheme, the identity authentication of the corresponding participant can be accurately realized. Store the voiceprint features of each participant, the target expression in the target scheme, and the average face features of each target expression. Identity authentication based on the voiceprint features and the average face features of each target expression can reduce the storage pressure while ensuring the accuracy of identity authentication and realize the effective management of participant information. Brief Description of the Drawings

[0024] Figure 1 is a flowchart of a method for managing participant information according to an embodiment of the present application.

[0025] Figure 2 is a structural block diagram of a system for managing participant information according to an embodiment of the present application. Detailed Embodiments

[0026] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0027] According to a first aspect of the present application, the present application provides a method for managing participant information, which is used to screen and store the participant information of all participants so that when participants join an online meeting, they can quickly and accurately complete identity authentication.

[0028] Figure 1It is a flowchart of a method for managing information of participants according to an embodiment of the present application. As Figure 1 shown, the method for managing information of participants includes steps S101 to S105, which are described in detail below.

[0029] S101, collect face images of multiple preset expressions of any participant and audio information of a preset statement.

[0030] In one embodiment, the participant is any participant in an online meeting system. In an enterprise, the participants include all employees within the enterprise; the preset expressions include multiple expressions such as opening the mouth, blinking, smiling, or raising the eyebrows. Collect face images of any participant under each preset expression, and collect the audio information when the participant reads the preset statement.

[0031] Among them, the preset statement is set in advance and is used to reflect the voiceprint characteristics of the participant, and there is no limitation here.

[0032] In this way, if the number of preset expressions is denoted as N, then one participant can collect N face images and a string of audio information. The N face images and a string of audio information are used for identity authentication when joining an online meeting. However, directly storing the N face images and audio information of all participants will occupy a large amount of storage resources. Therefore, it is necessary to screen these participant information.

[0033] S102, divide the audio information into audio segments of each character in the preset statement, use a long short-term memory network to obtain the audio features of each character, and weighted sum the audio features of each character according to the variance of the audio features of each participant to obtain the voiceprint characteristics of the participant.

[0034] In one embodiment, the preset statement includes multiple characters, and the audio information is divided to obtain the audio segment corresponding to each character. Specifically, dividing the audio information into audio segments of each character in the preset statement includes: performing frame division processing on the audio information and calculating the short-term energy at any time stamp to obtain a short-term energy sequence, dividing the short-term energy sequence according to the extreme points to obtain multiple time intervals; in response to the number of the time intervals being equal to the number of characters in the preset statement, taking the audio information within each time interval as the audio segment corresponding to the corresponding character, otherwise, re-collecting the audio information of the participant.

[0035] Among them, the frame division window at any time stamp is a time period centered on the time stamp and including multiple time stamps on both adjacent sides. Performing frame division processing on the audio information is a process of obtaining the frame division window of each time stamp.

[0036] Short-time energy is used to measure the energy intensity of an audio signal within a time window, which is a commonly used technique in the field of speech signal processing and will not be elaborated here. When the number of time intervals is equal to the number of characters in the preset statement, a one-to-one correspondence between time intervals and characters is achieved; when the number of time intervals is not equal to the number of characters in the preset statement, it indicates that there are missing characters or repeated readings in the audio information. At this time, the audio information of this participant needs to be re-collected to ensure that the audio segments of each character of this participant can be obtained.

[0037] In one embodiment, obtaining the audio features of each character by using a long short-term memory network includes: inputting the audio information into the long short-term memory network to obtain the short-term vectors of each timestamp; taking the average value of the short-term vectors of all timestamps within the audio segment of any character as the audio feature of the character.

[0038] Among them, the audio information is a time series data. The long short-term memory network can extract the long-term vector and short-term vector of each timestamp in the time series data. The long-term vector is used to represent the time series characteristics of all time series data from the start of the time series data to the corresponding timestamp, and the short-term vector is used to represent the time series characteristics of the corresponding timestamp in the time series data. It should be noted that the training process of the long short-term memory network is a well-known technology to those skilled in the art and will not be elaborated here.

[0039] Since different participants may have different pronunciations when reading the same character, when all participants read a character, the greater the difference among the participants, the more accurately the audio feature corresponding to the character can reflect the voiceprint characteristics of each participant. Therefore, the voiceprint characteristics of each participant are calculated based on the variance of the audio features of each participant.

[0040] Specifically, the voiceprint characteristics of the participant are:

[0041] , is the audio feature of the participant at the character , is the variance of the audio features of the character for all participants, is the sum of the variances of the audio features of all characters for all participants, is the number of characters in the preset statement.

[0042] In this way, the voiceprint characteristics of each participant are obtained.

[0043] S103. Extract features from the face images to obtain the face features of each preset expression, initialize the combination schemes of the preset expressions of each participant, and calculate the global significance of the combination scheme and the individual significance of each participant according to the average face features of the selected preset expressions in the combination scheme. The individual significance is the minimum difference in the average face features between a participant and other participants, and the global significance is negatively correlated with the variance of the individual significance.

[0044] In one embodiment, the face features corresponding to each face image can be extracted according to an autoencoder network or a principal component analysis algorithm. Since a participant can capture face images of multiple preset expressions, the face features of the participant under each preset expression can be obtained.

[0045] The combination scheme is a multi-dimensional 01 vector. If the value of any dimension is 1, it means that the preset expression corresponding to this dimension is selected; otherwise, the preset expression corresponding to this dimension is not selected. The number of preset expressions is N, and the dimension of the combination scheme is also N. One dimension in the combination scheme corresponds to one preset expression. For example, if the number of preset expressions is recorded as 5, and the combination scheme is , it means that the face features of the first preset expression and the fourth preset expression are selected.

[0046] In one embodiment, the combination schemes of each participant are initialized. One combination scheme corresponds to a selection situation of a preset expression. Under the combination schemes of each participant, the global significance and the individual significance of each participant are calculated. The individual significance is the minimum difference in face features between a participant and other participants. Taking the participant as an example, the greater the minimum difference in face features between the participant and other participants, the more significant the face features of the participant , which is more conducive to the identity recognition of the participant , and the greater the individual significance of the participant . The global significance is used to evaluate the degree of difference between the individual significances of all participants. When the individual significances of all participants are basically the same, all participants can obtain accurate identity recognition results. At this time, the global significance is relatively large.

[0047] Specifically, the average face feature of the participant is: : , is the face feature of the th selected preset expression in the combination scheme of the participant , is the participant The number of preset expressions selected in the combination scheme.

[0048] After obtaining the average facial features of all participants, the individual salience of each participant can be calculated. The calculation process of the individual salience of a participant is as follows: Calculate the Euclidean distance between the average facial features of the participant and other participants, and take the minimum Euclidean distance as the individual salience of the participant

[0049] The global salience is negatively correlated with the variance of the individual salience. The global salience satisfies the relationship: is the variance of the individual salience.

[0050] In this way, initialize the combination scheme of all participants. This combination scheme includes the selection of preset expressions by each participant, and the combination schemes of each participant are different. Under the combination schemes of all participants, calculate the individual salience of each participant, and the individual salience is used to evaluate the difficulty of identity authentication of each participant; at the same time, calculate the global salience of all participants, which is used to evaluate the difference degree between the individual saliences of all participants.

[0051] S104, taking the sum of the global salience and the minimum value of the individual salience as the objective function, update the combination scheme of each participant, and take the combination scheme corresponding to the maximum value of the objective function as the target scheme.

[0052] In one embodiment, the minimum value of the individual salience can reflect the highest difficulty of identity authentication of all participants. The larger the minimum value of the individual salience, the smaller the highest difficulty, which means that the difficulty for all participants to complete identity authentication is smaller. Therefore, take the minimum value of the individual salience as part of the objective function; further, the larger the global salience, the more consistent the difficulty of identity authentication of all participants. Taking the sum of the minimum value of the individual salience and the global salience as the objective function, when the value of the objective function is larger, it means that the difficulty for all participants to complete identity authentication is at an approximately small value, and all participants can obtain accurate identity authentication results.

[0053] After defining the objective function, taking the maximum value of the objective function as the optimization goal, continuously update the combination scheme of each participant until the maximum value of the objective function is reached and stop updating. Take the combination scheme corresponding to each participant at this time as the target scheme of the corresponding participant. Specifically, the particle swarm optimization algorithm, simulated annealing algorithm or genetic algorithm can be used to update the combination scheme of each participant until the target scheme of each participant is obtained.

[0054] ​​In this way, the target solutions of each participant are determined. A target solution of a participant includes at least one target expression, and the identity authentication of the corresponding participant can be accurately achieved according to the facial features corresponding to the target expression.

[0055] S105, store the voiceprint features of each participant, the target expressions in the target solution, and the average facial features of each target expression to implement the management of participant information. The average facial features and voiceprint features of each target expression are used for identity authentication.

[0056] In one embodiment, one is used to correspond to one target solution, and one target solution includes at least one target expression. To ensure the accuracy of identity authentication while reducing the information storage pressure, it is not necessary to store the facial images of all preset expressions and the audio information of preset statements. Only the voiceprint features, target expressions, and average facial features of each target expression of the participants need to be stored to implement the management of participant information. At the same time, using the voiceprint features and the average facial features of each target expression, the identity authentication when joining an online meeting is achieved by integrating the two dimensions of image features and audio features.

[0057] Specifically, the process of the identity authentication includes: in response to any participant receiving an invitation to join the meeting, sending an action instruction to the participant, where the action instruction includes the target expression of the participant; collecting the real-time voiceprint features and the real-time facial features of each target expression, and calculating the real-time average facial feature; calculating the facial similarity between the real-time average facial feature and the average facial feature, and the voiceprint similarity between the real-time voiceprint feature and the voiceprint feature. In response to both the facial similarity and the voiceprint similarity being greater than a preset similarity, the identity authentication is successful; otherwise, the identity authentication fails.

[0058] Among them, the preset similarity is 0.8.

[0059] It can be understood that since the target solutions of each participant are different, the action instructions of each participant are also different. In the process of identity authentication, it is only necessary to collect the facial images under the target expressions, avoiding the collection of facial images under all preset expressions, improving the authentication efficiency, and reducing the calculation amount during identity authentication.

[0060] According to the second aspect of the present application, the present application also provides a participant information management system. Figure 2 It is a structural block diagram of a participant information management system according to an embodiment of the present application. As Figure 2As shown, the system 50 includes a processor and a memory, and the memory stores computer program instructions, which implement a method for managing participant information according to the first aspect of the present application when executed by the processor. The system also includes a communication bus, a communication interface, and other components well-known to those skilled in the art, and their settings and functions are known in the art, so they will not be elaborated herein.

[0061] It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all fall within the protection scope of the present application.

Claims

1. A method for managing information of meeting participants, characterized in that: The management method comprises: Collect facial images of multiple preset expressions and audio information of preset sentences of any participant; Divide the audio information into audio segments of each character in the preset sentence, use the long short-term memory network to obtain the audio features of each character, and perform weighted summation of the audio features of each character according to the variance of the audio features of each participant to obtain the voiceprint features of the participant; Extract features from the face image to obtain the facial features of each preset expression, initialize the combination scheme of the preset expressions of each participant, and calculate the global significance of the combination scheme and the individual significance of each participant based on the average facial features of the selected preset expressions in the combination scheme; The global significance Satisfies the relationship: , is the variance of individual significance; The participants The calculation process of the individual significance of the participants is as follows: The Euclidean distance between the average facial features of other participants, and the minimum Euclidean distance is taken as the participant The individual significance of Taking the sum of the global significance and the minimum individual significance as the objective function, update the combination plan of each participant, and take the combination plan corresponding to the maximum value of the objective function as the target plan; The voiceprint features of each participant, the target expression in the target plan and the average facial features of each target expression are stored to achieve information management of the participants. The average facial features and voiceprint features of each target expression are used for identity authentication.

2. A method for managing information of participants according to claim 1, characterized in that: The audio segments that divide the audio information into the characters in the preset sentence include: The audio information is framed and the short-time energy of any time stamp is calculated to obtain a short-time energy sequence, and the short-time energy sequence is divided according to the extreme value points to obtain multiple time intervals; In response to the number of the time intervals being equal to the number of characters in the preset sentence, the audio information in each time interval is used as the audio segment of the corresponding character; otherwise, the audio information of the conference participants is recollected.

3. A method for managing information of participants according to claim 1, characterized in that: The audio features of each character obtained using the long short-term memory network include: Input the audio information into the long short-term memory network to obtain the short-term vector of each time stamp; The average value of all time stamp short-time vectors in the audio segment of any character is taken as the audio feature of the character.

4. A method for managing information of participants according to claim 1, characterized in that: Participants Voiceprint features for: , For participants In character The audio characteristics of Characters for all participants The audio feature variance is is the sum of the audio feature variances of all characters of all participants, The number of characters in the preset sentence.

5. A method for managing information of participants according to claim 1, characterized in that: The combination scheme is a multi-dimensional 01 vector. When the value of any dimension is 1, it indicates that the preset expression corresponding to the dimension is selected. Otherwise, the preset expression corresponding to the dimension is not selected.

6. A method for managing information of participants according to claim 1, characterized in that: Extracting features from facial images includes: extracting features from facial images based on an autoencoder network or a principal component analysis algorithm.

7. A method for managing participant information according to any one of claims 1 to 6, characterized in that: The identity authentication process includes: In response to any participant receiving a meeting invitation, issuing an action instruction to the participant, wherein the action instruction includes a target expression of the participant; Collect real-time voiceprint features and real-time facial features of each target expression, and calculate real-time average facial features; The face similarity between the real-time average face feature and the average face feature, and the voiceprint similarity between the real-time voiceprint feature and the voiceprint feature are calculated. If both the face similarity and the voiceprint similarity are greater than the preset similarity, the identity authentication is successful. Otherwise, the identity authentication fails.

8. A participant information management system, characterized in that: It comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a method for managing information of participants according to claim 7 is implemented.

Images