Conversation participant identification method and system based on intelligent glasses

Abstract

The invention relates to a dialogue participant identification method and system based on intelligent glasses. The intelligent glasses are provided with a bone conduction microphone and an air conduction microphone; the bone conduction microphone is arranged at a position in contact with the body of a wearer in a wearing state and is used for collecting bone conduction sound signals; the air conduction microphone collects an air conduction sound signal. According to the dialogue participant identification method based on the intelligent glasses, the bone conduction sound signal is used as the wearer sound signal, the wearer sound signal is subtracted from the air conduction sound signal, and the participant sound signal is obtained. The same dialogue participant recognition effect can be achieved without operating a neural network voiceprint recognition model with many parameters and a complex structure, and the method has the advantages of being small in computing power consumption, short in recognition time, good in real-time performance, high in recognition accuracy and high in cost performance.

Description

Technical Field

[0001] The present invention relates to the technical field of smart glasses, and in particular to a method and system for identifying conversation participants based on smart glasses. Background Art

[0002] Smart glasses are a type of wearable device that combines the visual assistance function of traditional glasses with the interaction and computing capabilities of modern smart devices. By equipping smart glasses with displays, cameras, microphones and other devices, they can provide users with a variety of functions such as information display, augmented reality (AR), photography, audio and video recording, navigation, and health management.

[0003] There is a kind of smart glasses equipped with a microphone in the prior art. In a multi-person dialogue scenario, the microphone of the smart glasses can record the sound signal generated by the multi-person dialogue, and then transmit the sound signal to the processor of the smart glasses or the host computer. The processor can distinguish different dialogue participants by running the dialogue participant identification method, so as to subsequently identify and record the speech content of different dialogue participants separately. This technology has significant practical significance in application scenarios such as conference recording.

[0004] However, existing methods for identifying conversation participants generally distinguish different conversation participants by analyzing the voiceprint of the voice. Since analyzing voiceprints requires relying on a neural network recognition model with more parameters and a more complex structure, existing methods for identifying conversation participants have the defects of high computing power consumption, long recognition time, and poor real-time performance. In addition, a person's voiceprint is not static. The voiceprint of the same person varies greatly in different physical states. Therefore, methods for identifying conversation participants based on voiceprints often make misjudgments and have low recognition accuracy. Summary of the invention

[0005] Based on this, the object of the present invention is to provide a pair of smart glasses, a method and a system for identifying conversation participants.

[0006] The present invention provides a pair of smart glasses, which are provided with a bone conduction microphone and an air conduction microphone; the bone conduction microphone is arranged at a position in contact with the body of the wearer in a wearing state, and collects bone conduction sound signals; the air conduction microphone collects air conduction sound signals.

[0007] Furthermore, the number of the air conduction microphones is ≧2; and the plurality of air conduction microphones respectively collect a plurality of air conduction sound signals.

[0008] Furthermore, the bone conduction microphone is arranged on the inner side of the left nose pad, the inner side of the right nose pad, the inner side of the left temple and / or the inner side of the right temple of the smart glasses.

[0009] Furthermore, each of the air conduction microphones is arranged at a position that does not contact the wearer's body when worn.

[0010] Furthermore, the number of the air conduction microphones is 3, which are respectively recorded as a first air conduction microphone, a second air conduction microphone and a third air conduction microphone; the first air conduction microphone is arranged on the left side of the frame of the smart glasses; the second air conduction microphone is arranged on the right side of the frame of the smart glasses; the third air conduction microphone is arranged in the middle part of the frame of the smart glasses.

[0011] Based on the smart glasses of the present invention, the present invention also provides a method for identifying conversation participants, including the steps of: S0, acquiring a bone conduction sound signal and an air conduction sound signal; S1, using the bone conduction sound signal as a wearer's sound signal; S2, subtracting the wearer's sound signal from the air conduction sound signal to obtain a participant's sound signal.

[0012] Furthermore, the number of the air-conducted sound signals is ≧2; step S2 specifically comprises: subtracting the wearer's sound signal from each of the air-conducted sound signals to obtain a plurality of mixed sound signals corresponding to the plurality of air-conducted sound signals respectively; and also comprises the following steps after step S2: S3, calculating the relative position of each conversation participant according to the phase difference of the plurality of mixed sound signals; and S4, distinguishing the sound signal of each conversation participant according to the relative position of each conversation participant.

[0013] Furthermore, step S3 specifically includes: S31, performing pairwise cross-correlation operations on multiple mixed sound signals to obtain a cross-correlation function of every two mixed sound signals; S32, extracting the peak of the cross-correlation function and obtaining the time coordinate of the peak point; calculating the phase difference between every two mixed sound signals according to the time coordinate of the peak point; S33, calculating the angle between the line connecting every two air conduction microphones and the position of the dialogue participant according to the phase difference between every two mixed sound signals; S34, locating the relative position of each dialogue participant according to the fixed geometric arrangement of the air conduction microphones and the angle between the line connecting every two air conduction microphones and the position of the dialogue participant.

[0014] Furthermore, step S31 specifically includes: S311, performing fast Fourier transform on each mixed sound signal to obtain the spectrum corresponding to each mixed sound signal; S312, performing pairwise multiplication operation on multiple spectra to obtain the multiplication result of each two spectra; S313, performing inverse fast Fourier transform on the multiplication result of each two spectra to obtain the cross-correlation function of each two mixed sound signals.

[0015] Based on the same inventive concept, the present invention also provides a conversation participant identification system, which includes: the above-mentioned smart glasses and a processor; the smart glasses collect bone conduction sound signals and air conduction sound signals through a bone conduction microphone and an air conduction microphone respectively; the processor distinguishes the wearer's sound signal and the participant's sound signal according to the bone conduction sound signal and the air conduction sound signal; wherein the processor performs the following steps: S1, taking the bone conduction sound signal as the wearer's sound signal; S2, subtracting the wearer's sound signal from the air conduction sound signal to obtain the participant's sound signal.

[0016] The present invention optimizes the types and arrangements of microphones mounted on smart glasses, allowing the bone conduction microphone and the air conduction microphone to collect audio signals from two sound propagation channels respectively. The wearer of smart glasses and other conversation participants can be quickly distinguished through a simple channel subtraction step, without the need to run a neural network voiceprint recognition model with many parameters and a complex structure. After distinguishing the wearer of smart glasses and other conversation participants, the present invention can distinguish different other conversation participants through a sound source localization method, also without the need to run a voiceprint recognition model. The conversation participant identification method of the present invention not only has low computing power consumption, short recognition time, and good real-time performance, but is also not affected by changes in the voiceprint of the speaker, has a high recognition accuracy, and has a high cost-effectiveness.

[0017] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the modules of the conversation participant identification system of the present invention; Figure 2 A schematic diagram of the positions of a bone conduction microphone and an air conduction microphone of smart glasses according to an embodiment of the present invention; Figure 3 This is a flow chart of a method for identifying a conversation participant according to Embodiment 1 of the present invention; Figure 4 This is a flowchart of a method for identifying a conversation participant according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0019] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0020] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0021] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0022] It should be understood that the embodiments of the present application are not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the embodiments of the present application is limited only by the appended claims.

[0023] The application scenario of the present invention is a multi-person conversation scenario. The conversation participants in the multi-person conversation scenario include: the wearer of the smart glasses, and one or more other conversation participants. The microphones mounted on existing smart glasses generally only have air conduction microphones, and the air conduction microphones receive the voices of all conversation participants equally. Therefore, the existing conversation participant identification method requires a complex voiceprint recognition method to accurately distinguish the voices of different conversation participants.

[0024] The present invention optimizes the types and arrangements of microphones mounted on smart glasses, allowing the bone conduction microphone and the air conduction microphone to collect audio signals from two sound propagation channels respectively. The wearer of smart glasses and other conversation participants can be quickly distinguished through a simple channel subtraction step, without the need to run a neural network voiceprint recognition model with many parameters and a complex structure. After distinguishing the wearer of smart glasses and other conversation participants, the present invention can distinguish different other conversation participants through a sound source localization method, also without the need to run a voiceprint recognition model. The conversation participant identification method of the present invention not only has low computing power consumption, short recognition time, and good real-time performance, but is also not affected by changes in the voiceprint of the speaker, has a high recognition accuracy, and has a high cost-effectiveness.

[0025] See also Figure 1 , Figure 1 It is a module diagram of the conversation participant identification system of the present invention.

[0026] The conversation participant identification system of the present invention comprises: smart glasses 100 and a processor 200. The smart glasses 100 are provided with a bone conduction microphone 110 and an air conduction microphone 120. The bone conduction microphone 110 collects bone conduction sound signals. The air conduction microphone 120 collects air conduction sound signals. The processor 200, according to the bone conduction sound signals and the air conduction sound signals, executes the conversation participant identification method of the present invention, thereby distinguishing the wearer sound signal generated by the speech of the smart glasses wearer and the participant sound signals generated by the speech of other conversation participants.

[0027] In this embodiment, the processor 200 is independent of the smart glasses 100. In other embodiments, the processor 200 may also be disposed inside the smart glasses 100 as a part of the smart glasses 100. The present invention does not limit the relationship between the smart glasses 100 and the processor 200.

[0028] See also Figure 2 , Figure 2 This is a schematic diagram of the positions of the bone conduction microphone and the air conduction microphone of the smart glasses according to an embodiment of the present invention.

[0029] The bone conduction microphone 110 is arranged on the smart glasses 100 at a position in contact with the wearer's body when the glasses are worn. Preferably, in this embodiment, the number of the bone conduction microphones 110 arranged on the smart glasses 100 is 4, which are respectively recorded as the first bone conduction microphone 111, the second bone conduction microphone 112, the third bone conduction microphone 113 and the fourth bone conduction microphone 114. The first bone conduction microphone 111 and the second bone conduction microphone 112 are respectively arranged on the inner sides of the left and right nose pads of the smart glasses 100, in contact with the nose bridge of the wearer. The third bone conduction microphone 113 and the fourth bone conduction microphone 114 are respectively arranged on the inner sides of the left and right temples of the smart glasses 100, in contact with the ears and / or temples of the wearer. In other embodiments, the number of the bone conduction microphones 110 may not be limited to 4, and each of the bone conduction microphones 110 may be distributed at a position in contact with the wearer's body in any wearing state.

[0030] The bone conduction microphone 110 collects bone conduction sound signals by sensing the slight vibration of the head and neck bones caused by a person speaking. Since the bone conduction microphone 110 is in contact with the wearer's body, when the wearer speaks, the voice of the wearer's speech will be conducted through the body to the bone conduction microphone 110, so that the bone conduction microphone 110 collects the wearer's speech. Since other conversation participants except the wearer do not contact the bone conduction microphone 110, the bone conduction sound signals collected by the bone conduction microphone 110 only include the speech of the wearer of the smart glasses, and do not include the speech of other conversation participants.

[0031] The air conduction microphone 120 collects air conduction sound signals by sensing the air vibration caused by the propagation of sound. Since air is everywhere, the air conduction microphone 120 can be set at any position on the smart glasses 100 in principle. In a multi-person conversation scenario where the number of other conversation participants other than the wearer is ≧2, in order to distinguish different conversation participants, the number of the air conduction microphones 120 should be ≧2. In this embodiment, the number of the air conduction microphones 120 is 3, which are respectively recorded as the first air conduction microphone 121, the second air conduction microphone 122 and the third air conduction microphone 123. In order to avoid occupying the position of the bone conduction microphone 110, the air conduction microphone 120 of this embodiment is preferably set at a position that does not contact the wearer's body in the wearing state. Specifically, the first air conduction microphone 121 and the second air conduction microphone 122 are respectively set on the left and right sides of the frame of the smart glasses 100, and the third air conduction microphone 123 is set in the middle of the frame of the smart glasses 100.

[0032] Furthermore, the air conduction microphone 120 is an omnidirectional air conduction silicon microphone. An omnidirectional air conduction silicon microphone is a miniature microphone made of silicon material, having omnidirectional sound pickup characteristics, and converting sound into electrical signals through the air conduction principle. It is particularly suitable for various application scenarios that require a wide sound pickup range, miniaturization, integration, and mass production.

[0033] Furthermore, the conversation participant identification system further includes a signal conditioning circuit, which includes but is not limited to modules such as a DC bias circuit, a filter circuit, a signal amplification circuit, and an analog-to-digital conversion circuit. The signal conditioning circuit is arranged at the front end of the processor 200, and is used to perform filtering, amplification, analog-to-digital conversion, and other preprocessing on the bone conduction sound signal and the air conduction sound signal, and then transmit the preprocessed bone conduction sound signal and the air conduction sound signal to the processor 200. The processor 200 performs the conversation participant identification method of the present invention according to the preprocessed bone conduction sound signal and the air conduction sound signal. In this embodiment, the signal conditioning circuit is arranged inside the smart glasses 100; in order to adapt to the structure of the smart glasses 100, the bone conduction microphone 110 and the air conduction microphone 120 are both connected to the signal conditioning circuit through a flexible printed circuit (Flexible Printed Circuit, FPC).

[0034] Example 1 In this embodiment, there is only one other conversation participant besides the smart glasses wearer, and there is no need to distinguish between different other conversation participants. Therefore, the conversation participant identification method of this embodiment only needs to distinguish between the speaking voices of the smart glasses wearer (corresponding to the wearer's voice signal) and other conversation participants (corresponding to the participant's voice signal).

[0035] See also Figure 3 , Figure 3 This is a flowchart of a method for identifying a conversation participant according to Embodiment 1 of the present invention.

[0036] The method for identifying a conversation participant executed by the processor 200 of Embodiment 1 of the present invention comprises the steps of: S0, obtaining bone conduction sound signals and air conduction sound signals. Since this embodiment does not distinguish the needs of different other conversation participants, the number of air conduction microphones 120 carried by the smart glasses 100 can be only one, and the number of corresponding air conduction sound signals can also be only one.

[0037] S1, using the bone conduction sound signal as the wearer's sound signal. From the working principle of the bone conduction microphone, it can be seen that the bone conduction sound signal only includes the wearer's speaking voice, so step S1 directly uses the bone conduction sound signal as the wearer's sound signal.

[0038] S2, subtract the wearer's voice signal from the air-conducted voice signal to obtain a participant voice signal. The air-conducted voice signal includes the voices of all conversation participants (including the wearer and other conversation participants). Therefore, the voice signal obtained by subtracting the wearer's voice signal from the air-conducted voice signal is a participant voice signal that mainly includes the voices of other conversation participants after removing the wearer's voice.

[0039] After obtaining the wearer's voice signal and the participant's voice signal, content recognition is performed on the wearer's voice signal and the participant's voice signal respectively, so that the speech contents of different dialogue participants can be obtained respectively.

[0040] Example 2 In this embodiment, the number of other conversation participants other than the smart glasses wearer is ≥ 2, so the conversation participant identification method needs to distinguish the wearer's voice signal generated by the smart glasses wearer speaking, and multiple participant voice signals generated by each different conversation participant speaking. Therefore, the difference between the embodiment 2 of the present invention and the embodiment 1 is: The number of air conduction microphones 120 mounted on the smart glasses 100 is ≧2, so that the number of corresponding air conduction sound signals is ≧2. The method for identifying a conversation participant executed by the processor 200 of embodiment 2 of the present invention comprises the steps of: S0, obtaining bone conduction sound signals and multiple air conduction sound signals.

[0041] S1, making the bone conduction sound signal as the wearer's sound signal.

[0042] S2, subtract the wearer's sound signal from each of the air-conducted sound signals to obtain a plurality of mixed sound signals corresponding to the plurality of air-conducted sound signals.

[0043] S3, calculating the relative position of each conversation participant according to the phase difference of the multiple mixed sound signals.

[0044] Each mixed sound signal is a mixture of the voices of multiple other dialogue participants. For one other dialogue participant, the path lengths that his voice travels from the dialogue participant's voice position to different air conduction microphones are different, and the speed of sound propagation in the air is fixed at about 300m / s. Therefore, different air conduction microphones receive different voices of different other dialogue participants at different time points (i.e., phases). By comparing the phase differences of multiple mixed sound signals, combined with the known positions of multiple air conduction microphones (different air conduction microphones are about 10cm apart), the voice position of the dialogue participant can be located through geometric relationship calculations, thereby distinguishing each different other dialogue participant based on the relative positions of multiple dialogue participants.

[0045] Specifically, step S3 includes: S31, performing a pairwise cross-correlation operation on the multiple mixed sound signals to obtain a cross-correlation function between each two mixed sound signals. The formula for performing a cross-correlation operation on two time signals is as follows: , where represents the cross-correlation function, represents the first time signal, represents the second time signal, The length of the sequence representing the time signal.

[0046] Furthermore, in order to improve the calculation speed, step S31 can also calculate the cross-correlation function by spectrum multiplication. At this time, step S31 specifically includes: S311, performing fast Fourier transform on each mixed sound signal to obtain the spectrum corresponding to each mixed sound signal; S312, performing pairwise multiplication operations on multiple spectra to obtain the multiplication result of each two spectra; S313, performing inverse fast Fourier transform on the multiplication result of each two spectra to obtain the cross-correlation function of each two mixed sound signals.

[0047] S32, extracting the peak value of the cross-correlation function and obtaining the time coordinate of the peak point; and calculating the phase difference between every two mixed sound signals according to the time coordinate of the peak point.

[0048] S33, calculating the angle between the line connecting each two air conduction microphones and the position of the dialogue participant according to the phase difference between each two mixed sound signals.

[0049] S34, locating the relative position of each dialogue participant according to the fixed geometric arrangement of the air conduction microphones and the angle between the line connecting every two air conduction microphones and the position of the dialogue participant.

[0050] S4, distinguishing each participant's voice signal according to the relative position of each dialogue participant. The voice signal of each participant obtained in step S4 includes the speaking voice of each other dialogue participant.

[0051] Subsequently, the wearer's voice signal and each participant's voice signal are respectively identified to obtain the speech content of different dialogue participants.

[0052] The present invention has the following technical effects: 1. The present invention optimizes the types and arrangements of microphones mounted on smart glasses, so that the bone conduction microphone and the air conduction microphone collect the audio signals of the two sound propagation channels respectively. The wearer of smart glasses and other conversation participants can be quickly distinguished through a simple channel subtraction step, without running a neural network voiceprint recognition model with many parameters and complex structure, and has the advantages of low computing power consumption, short recognition time, and good real-time performance. 2. After distinguishing the wearer of smart glasses from other conversation participants, the present invention can distinguish different other conversation participants through the sound source localization method, which not only has low computing power consumption and good real-time performance, but also will not be affected by the changes in the voiceprint of the speaker, and has a high recognition accuracy rate and a high cost performance.

[0053] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A method for identifying conversation participants based on smart glasses, wherein the smart glasses are provided with a bone conduction microphone and an air conduction microphone; the bone conduction microphone is arranged at a position in contact with the wearer's body in a wearing state to collect bone conduction sound signals; the air conduction microphone collects air conduction sound signals, characterized in that: Includes steps: S0, obtaining bone conduction sound signals and air conduction sound signals; S1, using the bone conduction sound signal as the wearer's sound signal; S2, subtract the wearer's sound signal from the air-conducted sound signal to obtain the participant's sound signal.

2. The method for identifying conversation participants based on smart glasses according to claim 1, characterized in that: The number of air conduction microphones of the smart glasses is ≧2; the number of air conduction sound signals is ≧2; Step S2 specifically includes: subtracting the wearer's sound signal from each of the air-conducted sound signals to obtain a plurality of mixed sound signals corresponding to the plurality of air-conducted sound signals; Also included are the steps after step S2: S3, calculating the relative position of each conversation participant according to the phase difference of the multiple mixed sound signals; S4, distinguishing the sound signal of each participant according to the relative position of each dialogue participant.

3. The method for identifying conversation participants based on smart glasses according to claim 2, characterized in that: Step S3 specifically includes: S31, performing a pairwise cross-correlation operation on a plurality of mixed sound signals to obtain a cross-correlation function of every two mixed sound signals; S32, extracting the peak value of the cross-correlation function and obtaining the time coordinate of the peak point; and calculating the phase difference between every two mixed sound signals according to the time coordinate of the peak point; S33, calculating the angle between the line connecting each two air conduction microphones and the position of the conversation participant according to the phase difference between each two mixed sound signals; S34, locating the relative position of each dialogue participant according to the fixed geometric arrangement of the air conduction microphones and the angle between the line connecting every two air conduction microphones and the position of the dialogue participant.

4. The method for identifying conversation participants based on smart glasses according to claim 3, characterized in that: Step S31 specifically includes: S311, performing fast Fourier transform on each mixed sound signal to obtain a frequency spectrum corresponding to each mixed sound signal; S312, performing a pairwise multiplication operation on the plurality of spectrums to obtain a multiplication result of every two spectrums; S313, performing inverse fast Fourier transform on the multiplication result of each two frequency spectra to obtain a cross-correlation function of each two mixed sound signals.

5. A conversation participant identification system, characterized in that: include: Smart glasses and processors; The smart glasses are provided with a bone conduction microphone and an air conduction microphone; the bone conduction microphone is arranged at a position in contact with the wearer's body when being worn, and collects bone conduction sound signals; the air conduction microphone collects air conduction sound signals; The processor distinguishes the wearer's voice signal and the participant's voice signal according to the bone conduction sound signal and the air conduction sound signal; wherein the processor performs the following steps: S1, taking the bone conduction sound signal as the wearer's voice signal; S2, subtracting the wearer's voice signal from the air conduction sound signal to obtain the participant's voice signal.

Images