Conference speaking management method based on mobile terminal, electronic equipment and storage medium

Abstract

The invention provides a conference speaking management method based on a mobile terminal, electronic equipment and a storage medium, the method is applied to a conference speaking management system, the conference speaking management system comprises the mobile terminal, a server and a loudspeaker, and the mobile terminal is provided with a microphone; the method comprises the following steps: at least when access request information sent by a mobile terminal is received, establishing a communication channel with the mobile terminal; when a first speaking instruction sent by the mobile terminal is received, obtaining initial audio information input by a microphone of the mobile terminal; and playing the user audio information in the initial audio information by using a loudspeaker. The method is helpful for improving the organization efficiency of the conference and the experience of participants, and has wide market application prospects and practical values.

Description

Technical Field

[0001] The present application relates to the technical field of conference communication, and in particular to a conference speech management method based on a mobile terminal, an electronic device and a storage medium. Background Art

[0002] In modern enterprises, meetings are widely used as a key means of communication and decision-making. In order to improve the efficiency and effectiveness of meetings, a combination system of speakers and microphones is usually used. In this system, the speaker speaks through the microphone, and the speaker is responsible for transmitting the speech content to all participants in the meeting room.

[0003] In the related art, wired microphones or wireless microphones are usually used as microphones. However, when there are many participants and they speak frequently, it is usually necessary to pass the microphone to the next speaker after the current speaker finishes speaking. The transfer process takes a long time, which seriously affects the efficiency and quality of the meeting. Summary of the invention

[0004] The present application is proposed in consideration of the above-mentioned problems. According to one aspect of the present application, a conference speech management method based on a mobile terminal is provided, and the method is applied to a conference speech management system, wherein the conference speech management system comprises a mobile terminal, a server and a speaker, and the mobile terminal has a microphone; the method comprises: establishing a communication channel with the mobile terminal at least when receiving an access request message sent by the mobile terminal; when receiving a first speech instruction sent by the mobile terminal, obtaining initial audio information inputted by the microphone of the mobile terminal; and using the speaker to play the user audio information in the initial audio information.

[0005] Exemplarily, the access request information is generated by the mobile terminal by executing the following steps: accessing the access interface of the server according to the service address of the server; and generating the access request information according to the identity information input by the user in the access interface.

[0006] Exemplarily, before executing the step of accessing the access interface of the server according to the service address of the server, the mobile terminal further executes the following step: scanning a QR code corresponding to the server to obtain the service address.

[0007] Exemplarily, the establishing of a communication channel with the mobile terminal at least upon receiving the access request information sent by the mobile terminal includes: upon receiving the access request information, establishing the communication channel with the mobile terminal; or, the access request information includes the identity information of the user, and the establishing of a communication channel with the mobile terminal at least upon receiving the access request information sent by the mobile terminal includes: upon receiving the access request information, comparing the identity information corresponding to the access request information with each identity information in a preset list, the preset list including at least one identity information; when the identity information corresponding to the access request information is the same as any identity information in the preset list, establishing the communication channel with the mobile terminal; or, the access request information includes the mobile phone number of the user, and the establishing of a communication channel with the mobile terminal at least upon receiving the access request information sent by the mobile terminal includes: upon receiving the access request information, sending first verification information to the mobile phone number corresponding to the access request information; upon receiving second verification information sent by the mobile terminal, comparing the second verification information with the first verification information; and when the second verification information is the same as the first verification information, establishing the communication channel with the mobile terminal.

[0008] Exemplarily, before using the speaker to play the user audio information in the initial audio information, the method also includes: performing acoustic echo elimination on the initial audio information at least based on the echo audio information to obtain user audio information, wherein the echo audio information is determined based on the last audio information played by the speaker; and sending the user audio information to the speaker to play the user audio information using the speaker.

[0009] Exemplarily, the performing acoustic echo elimination on the initial audio information at least based on the echo audio information to obtain the user audio information includes: processing the initial audio information based on the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, the feedback compensation coefficient and the echo audio information to obtain the user audio information.

[0010] Exemplarily, the processing the initial audio information based on the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, the feedback compensation coefficient, and the echo audio information to obtain the user audio information includes: processing the initial audio information through the following formula: e(n)=(T now -T ntl -T stl )×d(n)-cf×y(n) Where, e(n) represents the user audio information, T now Indicates the current server time, T stl represents the spatial transmission delay, T ntl represents the network delay, d(n) represents the initial audio information, cf represents the feedback compensation coefficient, and y(n) represents the echo audio information.

[0011] Exemplarily, after establishing a communication channel with the mobile terminal, the method further includes: receiving test information sent by the mobile terminal, the test information including first time information, the first time information being the time when the mobile terminal sends the test information; when receiving the test information, sending feedback information to the mobile terminal; obtaining second time information from the mobile terminal, the second time information being the time when the mobile terminal receives the feedback information; determining the network delay based on the difference between the second time information and the first time information; and / or, after establishing a communication channel with the mobile terminal, the method further includes: playing preset audio information using the speaker; receiving feedback audio information collected by the mobile terminal, the feedback audio information being obtained by collecting the preset audio information using the microphone of the mobile terminal; determining the spatial transmission delay based on the difference between the fourth time information and the third time information; wherein the third time information is the time when the preset audio information is sent, and the fourth time information is the time when the feedback audio information is received.

[0012] According to another aspect of the present application, an electronic device is provided, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the computer program to implement the above method.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, storing a computer program / instruction, and the computer program / instruction implements the above method when executed by a processor.

[0014] In the above technical solution, when at least the access request information sent by the mobile terminal is received, a communication channel is established with the mobile terminal, and when the first speech instruction sent by the mobile terminal is received, the initial audio information input by the microphone of the mobile terminal is obtained, thereby, the user's mobile terminal can be used as a virtual wireless microphone, and the user can directly use his own mobile terminal to speak without being restricted by the equipment in the venue, which can expand the applicability of the scene equipment. And the IP network of the mobile terminal can be directly used to replace the dedicated wired or wireless transmission line, which can expand the convenience of scene use. When the scheme is applied to application scenarios with many participants and frequent speaking needs, the problem of needing to transfer microphones in related technologies can be overcome, which helps to improve the meeting efficiency and meeting quality of large meetings. In short, the scheme can make full use of the popularity and convenience of mobile terminals by using the user's mobile terminal as a microphone, and improve the meeting efficiency and meeting quality. The scheme can be applied to various conference occasions where speeches are required, such as corporate annual meetings, academic seminars, government work meetings, etc. Whether it is a large-scale international conference or a small internal discussion, it can provide efficient and convenient speaking solutions, effectively solve many problems in the use of traditional conference microphones, improve the organizational efficiency of the meeting and the experience of the participants, and has broad market application prospects and practical value.

[0015] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 A schematic block diagram showing a traditional speaking system in the related art; Figure 2 A schematic flow chart showing a conference speech management method according to an embodiment of the present application; Figure 3 A schematic diagram showing audio information processing when the speaking mode is a mixed speaking mode according to an embodiment of the present application is shown; Figure 4 A schematic diagram showing audio information processing when the speaking mode is the host control mode according to an embodiment of the present application is shown; Figure 5 A schematic diagram showing audio information processing when the speaking mode is a post-speaking mode according to an embodiment of the present application is shown; Figure 6 A schematic diagram showing audio information processing when the speaking mode is a sequential mode according to an embodiment of the present application is shown; Figure 7 A schematic block diagram showing a conference speech management system according to an embodiment of the present application is shown; Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0019] As mentioned above, in the related art, wired microphones or wireless microphones are usually used as microphones. However, in conference environments such as large venues, there are many participants and the need to speak is frequent. The traditional wired microphone or wireless microphone transmission method has many inconveniences. First, the transmission efficiency of the microphone is low, especially in large venues. It takes a long time for the microphone to be passed from one speaker to the next. Secondly, the management and maintenance of the microphone is also relatively complicated, and it is easy to be lost or damaged during the transmission process. In addition, when the venue is large, the coverage of the microphone may be limited, resulting in some participants being unable to clearly hear the speech. These problems seriously affect the efficiency and quality of the meeting. In some large conference scenarios, the above problems are solved by using multiple wireless microphones to speak in turns, or by setting up multiple speaking points to disperse the microphones. The speaking system used in this method is as follows: Figure 1 As shown. Figure 1 In the wireless microphone, a dedicated receiver can be connected to the sound reinforcement system (i.e., the speaker) by wireless connection. Figure 1In the embodiment, the sound reinforcement system is composed of an audio processor, a power amplifier and a speaker) connected. The wired microphone is connected to the sound reinforcement system by a wired connection. When the method of taking turns to speak by using multiple wireless microphones is adopted, the number of wireless microphones is limited, and when there are a large number of speakers, the microphone still needs to be transferred. When the solution of setting multiple speaking points and setting wired microphones at the speaking points is adopted, due to the limitation of the number of speaking points, the speaker needs to move to the speaking point to speak, which also affects the efficiency of the meeting, and is limited by the complexity of the speaking point setting and the dependence on the layout of the venue, this method has great limitations. In short, the related art faces the problems of low meeting efficiency and poor quality when conducting large-scale meetings. In view of this, the present application provides a conference speech management method based on a mobile terminal, an electronic device and a storage medium, which helps to improve meeting efficiency and meeting quality. The method, electronic device and storage medium are described in detail below.

[0020] According to one aspect of the embodiment of the present application, a conference speech management method based on a mobile terminal is provided, and the method is applied to a conference speech management system. The conference speech management system includes a mobile terminal, a server and a speaker, the mobile terminal has a microphone, and the server can be used to implement the method described below. Figure 2 FIG. 1 is a schematic flow chart of a conference speech management method according to an embodiment of the present application. Figure 2 As shown, the method 200 may include the following steps S210, S220 and S230.

[0021] In step S210, at least when access request information sent by the mobile terminal is received, a communication channel with the mobile terminal is established.

[0022] In the solution of this example, each mobile terminal can correspond to each user one by one. That is, each user can send access request information to the server through his own mobile terminal, and the server can establish a communication channel with the mobile terminal when receiving at least the access request information of the mobile terminal, so as to exchange data. The mobile terminal includes but is not limited to any mobile terminal with a microphone, such as a mobile phone, a phone watch, and a tablet computer. In a specific embodiment, the mobile terminal can be a user's mobile phone.

[0023] It can be understood that there may be multiple mobile terminals. When there are multiple mobile terminals, a communication channel with the mobile terminal that sends the access request information may be established when at least one access request information sent by any mobile terminal is received.

[0024] In step S220, when the first speaking instruction sent by the mobile terminal is received, initial audio information input through the microphone of the mobile terminal is acquired.

[0025] It can be understood that the first speaking instruction can be an instruction for indicating that the user requests to speak. In some embodiments, the user can send the first speaking instruction through the mobile terminal when he needs to speak. In a specific embodiment, when the user needs to speak, he can click the start speaking button on the user interaction interface of the mobile terminal to make the mobile terminal send the first speaking instruction. Then, the user can use the microphone of the mobile terminal to speak, and the microphone of the mobile terminal can collect the user's speech and send it to the server.

[0026] In some embodiments, step S220, when receiving the first speaking instruction sent by the mobile terminal, obtaining the initial audio information input through the microphone of the mobile terminal, may specifically include the following steps: in response to the first speaking instruction, providing a data channel for transmitting audio information; and obtaining the initial audio information using the data channel. The data channel may be established through technologies such as WebRTC to achieve secure transmission of audio information between the mobile terminal and the server.

[0027] Optionally, before step S220, the method may further include the following steps: sending a speech prompt message to the mobile terminal, wherein the step of sending the first speech instruction by the mobile terminal is performed after the mobile terminal receives the speech prompt message. In this scheme, after receiving the speech prompt message, the user can choose whether to speak according to the speech prompt message. In some embodiments, the speech prompt message can be generated in response to the speech instruction input by the administrator (also referred to as the host). That is, the administrator can input the speech instruction through the control terminal (such as a server or a terminal connected to the server in communication) to select the current speaker. After receiving the speech instruction, the speech prompt message can be generated and sent to the mobile terminal of the target speaker specified by the speech instruction. The target speaker can choose whether to speak when the mobile terminal receives the speech prompt message. The above-mentioned sending method of the speech prompt message is only an example. In other embodiments, when the number of mobile terminals is multiple, the speech prompt message can be sent to each mobile terminal in turn according to the order in which each mobile terminal sends the access request message. The above-mentioned optional technical solution guides the user to start speaking through the speech prompt message, which can make the conference speech more orderly, thereby helping to further improve the quality of the conference.

[0028] Optionally, the method may further include the following steps: when receiving the second speech instruction sent by the mobile terminal, stop acquiring the initial audio information input through the microphone of the mobile terminal. In some embodiments, the start speech and end speech options may be displayed on the user interaction interface of the mobile terminal, and when the user clicks the start speech option, the mobile terminal generates the first speech instruction. When the user clicks the end speech option, the mobile terminal generates the second speech instruction. Of course, the user interaction interface of the mobile terminal may also include other options, for example, it may include a wait speech option. For example, in the above-mentioned embodiment in which the user can choose whether to speak according to the speech prompt information after receiving the speech prompt information, the user can choose the wait speech option when receiving the speech prompt information to generate and send the third speech instruction. When the third speech instruction is received, the speech prompt information can be sent to the next mobile terminal (the next mobile terminal can be determined according to the host's selection or according to the order in which the mobile terminal sends the access request information, which will not be repeated).

[0029] In step S230, the user audio information in the initial audio information is played by using a speaker.

[0030] After obtaining the initial audio information, the user audio information in the initial audio information can be played by the speaker. It is understandable that when the user's speech is collected by the microphone of the mobile terminal, the noise in the environment (that is, the audio information not belonging to the user's speech) may be collected together. Such noise includes but is not limited to the noise in the venue and the audio information currently played by the speaker.

[0031] Optionally, playing the user audio information in the initial audio information using a speaker may specifically include the following steps: sending the user audio information in the initial audio information to a decoding terminal for decoding, wherein the decoding terminal is used to send the user audio information to the speaker after decoding, so as to play the user audio information using the speaker.

[0032] Optionally, in step S230, before playing the user audio information in the initial audio information using the speaker, the method may further include the following steps: preprocessing the initial audio information to obtain the user audio information. The preprocessing method includes but is not limited to filtering, denoising, echo simulation, etc., which is not limited in this application.

[0033] In the above technical solution, when at least the access request information sent by the mobile terminal is received, a communication channel is established with the mobile terminal, and when the first speech instruction sent by the mobile terminal is received, the initial audio information input by the microphone of the mobile terminal is obtained, thereby, the user's mobile terminal can be used as a virtual wireless microphone, and the user can directly use his own mobile terminal to speak without being restricted by the equipment in the venue, which can expand the applicability of the scene equipment. And the IP network of the mobile terminal can be directly used to replace the dedicated wired or wireless transmission line, which can expand the convenience of scene use. When the scheme is applied to application scenarios with many participants and frequent speaking needs, the problem of needing to transfer microphones in related technologies can be overcome, which helps to improve the meeting efficiency and meeting quality of large meetings. In short, the scheme can make full use of the popularity and convenience of mobile terminals by using the user's mobile terminal as a microphone, and improve the meeting efficiency and meeting quality. The scheme can be applied to various conference occasions where speeches are required, such as corporate annual meetings, academic seminars, government work meetings, etc. Whether it is a large-scale international conference or a small internal discussion, it can provide efficient and convenient speaking solutions, effectively solve many problems in the use of traditional conference microphones, improve the organizational efficiency of the meeting and the experience of the participants, and has broad market application prospects and practical value.

[0034] Exemplarily, the access request information is generated by the mobile terminal by executing the following steps: accessing the access interface of the server according to the service address of the server; and generating the access request information according to the identity information input by the user in the access interface.

[0035] Optionally, the service address of the server can be directly provided by the conference organizer, or can be determined based on specific information (such as a QR code) provided by the conference organizer. After obtaining the service address, the user can enter the service address into a specific software (such as a browser) of the mobile terminal to access the access interface of the server. Multiple configuration items can be set on the access interface of the server, and multiple configuration items can be set according to actual needs, which will not be described in detail.

[0036] In a specific embodiment, the specific software is a browser in a mobile terminal. In this way, the user does not need to install additional applications on the mobile terminal, and can directly connect to the application using the system browser. And the IP network can be directly used without a dedicated network. This solution can directly access the access interface of the server through a cellular network (such as a 5G network, a 4G network, etc.) without connecting to a local network. It is easy to operate and helps to improve the user experience.

[0037] In this example, the user can enter identity information on the access interface, and the identity information can be used to identify the user. The identity information includes but is not limited to the current user's user name, mobile phone number, ID number, email address, etc. After the user enters the identity information, access request information can be generated based on the identity information and sent to the server.

[0038] The above technical solution can generate access request information based on the identity information input by the user, and the access request information can facilitate the identification and recognition of the identity of each user, which helps to provide a reliable basis for subsequent steps. For example, in the above embodiment where the administrator selects a speaker, the administrator can select a speaker from multiple users based on the user identity identified by the access request information. For another example, the access request information can provide a basis for conference security verification, prevent non-participants from joining the conference, and improve the security and confidentiality of the conference.

[0039] Exemplarily, before executing the step of accessing the access interface of the server according to the service address of the server, the mobile terminal further executes the following steps: scanning the QR code corresponding to the server to obtain the service address. In some embodiments, the QR code can be posted in multiple places at the conference site, or the QR code can be projected at the conference site. This method of obtaining the service address through the QR code can improve the convenience of use and facilitate users to obtain the service address of the server.

[0040] It is understood that when receiving the access request information of the mobile terminal, the communication channel with the mobile terminal can be directly established, or identity authentication can be performed according to the access request information, and then the communication channel with the mobile terminal can be established after the identity authentication is passed. The specific implementation methods of step S210 are introduced in three example schemes.

[0041] In an example solution, step S210, at least when receiving access request information sent by the mobile terminal, establishing a communication channel with the mobile terminal may include the following steps: when receiving access request information, establishing a communication channel with the mobile terminal. This solution can directly establish a communication channel with the mobile terminal when receiving access request information, which is convenient and fast, and is conducive to improving the efficiency of users joining the meeting. This example solution can be called a registration-based identity authentication solution.

[0042] In an example solution, the access request information includes the identity information of the user, and step S210, at least when the access request information sent by the mobile terminal is received, a communication channel with the mobile terminal is established, which may specifically include the following steps: when the access request information is received, the identity information corresponding to the access request information is compared with each identity information in a preset list, the preset list includes at least one identity information; when the identity information corresponding to the access request information is the same as any identity information in the preset list, a communication channel with the mobile terminal is established. This example solution may be called a list-based identity authentication solution.

[0043] Optionally, a preset list can be created as needed. For example, a preset list can be created in advance based on the information of all participants. The type of identity information stored in the preset list is the same as the type of information in the identity information corresponding to the access request information, for example, both can include a user name and a password.

[0044] The above technical solution can prevent people who are not on the list from joining the meeting by comparing the identity information corresponding to the access request information with each identity information in the preset list, which helps to improve the security and confidentiality of the meeting.

[0045] In an example solution, the access request information includes the user's mobile phone number. Step S210, at least when the access request information sent by the mobile terminal is received, a communication channel with the mobile terminal is established, which may include the following steps: when the access request information is received, first verification information is sent to the mobile phone number corresponding to the access request information; when the second verification information sent by the mobile terminal is received, the second verification information is compared with the first verification information; when the second verification information is the same as the first verification information, a communication channel with the mobile terminal is established. This example solution may be called a verification-based identity authentication solution.

[0046] It is understood that the first verification information may be in the form of a verification code. After receiving the first verification information, the user may input the first verification information into the mobile terminal, and the mobile terminal generates the second verification information based on the verification information input by the user, and sends the second verification information to the server so that the server can verify it.

[0047] The above-mentioned method of verifying the user identity through a verification-based identity authentication scheme and establishing a communication channel with the mobile terminal helps to improve the security and confidentiality of the meeting. In addition, this method of generating verification information in real time can avoid the inability to join the meeting due to the user forgetting the password, which helps to improve the user experience.

[0048] Exemplarily, the number of mobile terminals establishing the communication channel is multiple, and before step S220, the method may further include the following steps: determining the current speaking mode based on the mode selection instruction input by the target user. The target user may be the organizer or host of the meeting.

[0049] Optionally, the speaking mode may include a mixed speaking mode. When the current speaking mode is the mixed speaking mode, step S220 may specifically include the following steps: when receiving first speaking instructions respectively sent by multiple mobile terminals, respectively obtaining initial audio information respectively inputted by microphones of the multiple mobile terminals. Step S230 may specifically include the following steps: using a speaker to simultaneously play user audio information in each initial audio information.

[0050] In some embodiments, after respectively acquiring the initial audio information input by the microphones of multiple mobile terminals, the user audio information included in each of the multiple initial audio information can be sent to a decoding terminal, so that the decoding terminal can be used to decode the multiple user audio information, and the mixed user audio information (i.e., the speech of multiple users) can be output simultaneously through a speaker. Figure 3 FIG. 2 is a schematic diagram showing the audio information processing when the speech mode is a mixed speech mode according to an embodiment of the present application. Figure 3 As shown, when the speaking mode is the mixed speaking mode, the user audio information corresponding to the speaker 1, the speaker 2, ... the speaker N is simultaneously input to the decoding terminal and mixed and outputted by the speaker.

[0051] The above optional solution can output the speeches of multiple users simultaneously by setting the mixed speech mode. This mode can be suitable for scenarios where multiple people speak at the same time, such as meetings or seminars.

[0052] Optionally, the speech mode may include a moderator control mode. In some embodiments, before step S220, the method may further include the following steps: sending speech prompt information to the mobile terminal. In this optional implementation scheme, when the speech mode is the moderator control mode, the speech prompt information may be generated in response to the speech instruction input by the host. That is, the host can control the user's speech by sending speech prompt information to different mobile terminals. In other embodiments, step S220 may specifically include the following steps: when receiving the first speech instruction sent by each of the multiple mobile terminals, respectively obtain the initial audio information input by the microphones of the multiple mobile terminals. Step S230 may specifically include the following steps: according to the control instruction input by the target user, use the speaker to play the target user audio information in the multiple user audio information corresponding to each of the multiple initial audio information. For example, when the number of mobile terminals currently establishing a communication channel is N, the host can arbitrarily select a user from the users 1 to N corresponding to the N mobile terminals as the current speaker. This mode can improve the orderliness of the meeting and is suitable for meetings such as discussions and debates hosted by a clear host.

[0053] Figure 4 FIG. 2 is a schematic diagram showing the audio information processing when the speech mode is the host control mode according to an embodiment of the present application. Figure 4 As shown, the speakers include speaker 1, speaker X, ... speaker N. When user audio information corresponding to multiple speakers is simultaneously input into the decoding terminal, the host can use the control terminal to input a control instruction (for example, a control instruction for selecting X) so that the decoding terminal outputs the user audio information corresponding to speaker X to the speaker.

[0054] Optionally, the speaking mode may include a post-sent mode. The method may further include the following steps: when receiving a first speaking instruction sent by a new mobile terminal, stop acquiring the initial audio information input through the microphone of the current mobile terminal, and receive the initial audio information input through the microphone of the new mobile terminal. In this scheme, the new speaker is allowed to immediately take over the current speaker to speak. When a new speaker starts speaking, the initial audio information corresponding to the new speaker will replace the initial audio information corresponding to the current speaker and become the currently active audio stream. This mode is suitable for scenarios that require flexible switching of speakers, such as multi-party discussions or meetings where speakers take turns to speak.

[0055] Figure 5 FIG. 2 is a schematic diagram showing the audio information processing when the speaking mode is the last-sent mode according to an embodiment of the present application. Figure 5As shown, the speakers include speaker 1, speaker 2, ... speaker N. When the speaking mode is the post-speaking mode, when speaker N speaks, the audio information corresponding to speaker N will replace the audio information corresponding to the current speaker 1. At this time, the decoding terminal can output the user audio information corresponding to speaker N to the speaker.

[0056] Optionally, the speaking mode may include a sequential mode. Step S220 may specifically include the following steps: when a first speaking instruction is received from each of the multiple mobile terminals, the initial audio information input by the microphone of each mobile terminal is sequentially obtained according to a preset order. Among them, the step of obtaining the initial audio information input by the microphone of the next mobile terminal is performed when the microphone of the previous mobile terminal stops inputting the initial audio information. In the scheme of this embodiment, when there are multiple speakers, multiple speakers can speak in sequence in a preset order. The user audio information corresponding to each speaker is processed separately by the decoding terminal and then output. This mode is suitable for scenarios where speeches need to be listened to in a specific order, such as voting or reporting.

[0057] Figure 6 A schematic diagram of audio information processing when the speaking mode is a sequential mode according to an embodiment of the present application is shown. In the scheme of this embodiment, the speakers include speaker 1, speaker 2, ... speaker N. Multiple speakers can speak in sequence according to a preset order, and the user audio information corresponding to each speaker is decoded by the decoding terminal and transmitted to the speaker.

[0058] In the above technical solution, a suitable speaking mode can be selected in advance according to the needs of the meeting, so as to ensure that the current speaking mode is compatible with the meeting type, which helps to improve meeting efficiency and user experience.

[0059] In a specific embodiment, the mobile terminal can first obtain the service address by scanning the code, then open the browser, log in to the access interface according to the service address, and establish communication with the server according to the current identity authentication method (including registration, list and verification). Then, according to the current speaking mode and user needs, the first speaking instruction or the second speaking instruction can be sent to start or end the speech.

[0060] Exemplarily, in step S130, before using a speaker to play the user audio information in the initial audio information, the method may further include the following steps: performing acoustic echo elimination on the initial audio information at least based on the echo audio information to obtain the user audio information, wherein the echo audio information is determined based on the previous audio information played by the speaker; and sending the user audio information to the speaker to play the user audio information using the speaker.

[0061] In some implementations of this example, an estimate of the echo signal, i.e., echo audio information, can be generated by an estimated acoustic transfer function and based on the last audio information played by the speaker. The method of determining the echo audio information based on the last audio information played by the speaker is similar to the method of determining the echo audio information in the typical AEC technology, and will not be described in detail.

[0062] In some embodiments, performing acoustic echo cancellation on the initial audio information at least based on the echo audio information may include: subtracting the echo audio information from the initial audio information to obtain the user audio information. Of course, the initial audio information may also be acoustically echo cancelled in the manner described in the following embodiments, which will not be described in detail.

[0063] In the solution of this example, acoustic echo cancellation can be performed on the initial audio information based on the echo audio information. In this way, the part of the initial audio information that is irrelevant to the current speaker can be effectively removed, which helps to improve the audio quality played by the speaker and further helps to improve the quality of the meeting.

[0064] Exemplarily, at least based on the echo audio information, the initial audio information is acoustically echo eliminated to obtain the user audio information, which may specifically include the following steps: based on the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, the feedback compensation coefficient and the echo audio information, the initial audio information is processed to obtain the user audio information.

[0065] Optionally, the initial audio information may be delayed based on the network delay between the mobile terminal and the server and the spatial transmission delay between the mobile terminal and the server, and then the echo audio information processed by the feedback compensation coefficient may be subtracted to obtain the user audio information.

[0066] Optionally, the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, and the feedback compensation coefficient can be determined based on user experience, or by the method described in the following embodiments, which will not be repeated here.

[0067] The above technical solution fully considers the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, and the feedback compensation coefficient when performing acoustic echo cancellation, which helps to more accurately remove the part of the initial audio information that is not related to the current speaker.

[0068] Exemplarily, based on the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, the feedback compensation coefficient, and the echo audio information, the initial audio information is processed to obtain the user audio information, including: processing the initial audio information by the following formula: e(n)=(T now -Tntl -T stl )×d(n)-cf×y(n) In the formula, e(n) represents the user audio information, T now Indicates the current server time, T stl Represents the spatial transmission delay, T ntl represents network delay, d(n) represents initial audio information, cf represents feedback compensation coefficient, and y(n) represents echo audio information.

[0069] Optionally, the feedback compensation coefficient may be determined through a pre-test or based on user experience, which will not be described in detail.

[0070] Exemplarily, after establishing a communication channel with a mobile terminal, the method further includes the following steps: receiving test information sent by the mobile terminal, the test information including first time information, the first time information being the time when the mobile terminal sends the test information; sending feedback information to the mobile terminal upon receiving the test information; obtaining second time information from the mobile terminal, the second time information being the time when the mobile terminal receives the feedback information; determining network delay based on the difference between the second time information and the first time information; and / or, after establishing a communication channel with a mobile terminal, the method further includes: playing preset audio information using a speaker; receiving feedback audio information collected by the mobile terminal, the feedback audio information being obtained by collecting preset audio information using a microphone of the mobile terminal; determining spatial transmission delay based on the difference between fourth time information and third time information; wherein the third time information is the time when the preset audio information is sent, and the fourth time information is the time when the feedback audio information is received. For ease of description, the above-mentioned step of determining network delay can be referred to as an echo simulation network parameter initialization and correction operation, and the above-mentioned step of determining spatial transmission delay can be referred to as an echo simulation sound transmission delay initialization and correction operation.

[0071] As described above, the first time information is the time when the mobile terminal sends the test information, that is, the time displayed by the mobile terminal when sending the test information. The second time information is the time when the mobile terminal receives the feedback information, that is, the time displayed by the mobile terminal when receiving the feedback information. Similarly, the third time information is the time displayed by the server itself when the server sends the preset audio information, and the fourth time information is the time displayed by the server itself when receiving the feedback audio information.

[0072] Optionally, the method may further include the following steps: determining a feedback compensation coefficient based on the sound intensity and frequency of the preset audio information and the sound intensity and frequency of the feedback audio information.

[0073] For ease of understanding, the above-mentioned echo simulation network parameter initialization and correction operation and the echo simulation sound transmission delay initialization and correction operation are respectively explained below by way of examples.

[0074] In some embodiments, the echo simulation network parameter initialization and correction operation is implemented as follows: the mobile terminal sends a test message RT1 (ID, Timestap1) to the server, where ID is the connection number (i.e., the number of the communication channel); Timestap1 is the time of the mobile terminal, i.e., the first time information. After receiving RT1, the server sends a feedback message RT2 (ID, Timestap2) to the mobile terminal, where Timestap2 is the time displayed when the server sends RT2, and the time when the mobile terminal receives RT2 is Timestap3 (i.e., the second time information), and Timestap3 is the client time.

[0075] Among them, the network delay T ntl =Timestamp3-Timestamp1.

[0076] In the solution of this example, after obtaining the network delay, the method may further include: calibrating the local time of the mobile terminal according to the network delay, wherein: Local time after calibration = local time before calibration + (Timestape2-Timestape1)-T ntl / 2.

[0077] Optionally, the method may further include: periodically performing an echo simulation network parameter initialization and correction operation to correct the network delay. In the solution of this embodiment, the echo simulation network parameter initialization and correction operation may be performed at intervals of nT (e.g., 180 seconds) to correct T ntl .

[0078] In some embodiments, the implementation method of the echo simulated sound transmission delay initialization and correction operation is as follows: the server outputs a short-time specific audio signal (i.e., preset audio information, which can be a test audio signal of about 10ms generated by the server) to the decoding terminal, and transmits it to the speaker via the decoding terminal for output. The preset audio information can be recorded as Sp1 (Timestape1, frequency 1, sound intensity 1), that is, the sending time of the preset audio information is Timestap1 (i.e., the third time information), the frequency of the preset audio information is frequency 1, and the sound intensity is sound intensity 1. The speaker outputs the preset audio information, propagates through the on-site space, and reaches the mobile terminal. The mobile terminal collects the information through a microphone to obtain feedback audio information and transmits it back to the server. The feedback audio information obtained by the server can be recorded as Sp2 (Timestape2, frequency 2, sound intensity 2), that is, the time when the server receives the feedback audio information is Timestap2 (i.e., the fourth time information), the frequency of the feedback audio information is frequency 2, and the sound intensity is sound intensity 2.

[0079] Among them, the spatial transmission delay T stl =Timestape2-Timestape1-T ntl / 2.

[0080] In this embodiment, the preset audio information may adopt infrasonic or ultrasonic frequencies, and the duration and intensity of the preset audio information may be controlled to avoid interfering with the normal meeting.

[0081] In the solution of this embodiment, the feedback compensation coefficient cf can be determined by the following formula: cf=F(frequency 1, frequency 2, sound intensity 1, sound intensity 2).

[0082] Specifically, the feedback compensation coefficient cf can be calculated by taking the weighted average of the upper and lower intervals, which will not be elaborated herein.

[0083] Optionally, the method may further include: periodically performing an echo-simulated sound transmission delay initialization and correction operation, for example, performing an echo-simulated sound transmission delay initialization and correction operation once every time period nT (for example, 600 seconds) to correct T stl and cf.

[0084] The above technical solution can determine the network delay and / or spatial transmission delay more accurately, thereby providing a more reliable basis for the acoustic echo cancellation operation in the subsequent steps, which helps to further improve the quality of the obtained user audio information.

[0085] Figure 7 FIG. 1 is a schematic block diagram of a conference speech management system according to an embodiment of the present application. Figure 7 As shown, the conference speech management system includes a mobile terminal 710, a server 720, a local sound reinforcement system (ie, a speaker) 730, and a decoding terminal 740. The mobile terminal 710 has a built-in browser, and the server 720 is used to execute the conference speech management method of any of the above embodiments.

[0086] According to another aspect of the embodiments of the present application, an electronic device is also provided. Figure 8 FIG. 1 is a schematic block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device 800 includes: a processor 810 and a memory 820. The memory 820 stores a computer program, and the processor 810 is used to execute the computer program to implement the above method.

[0087] According to another aspect of the embodiment of the present application, a computer-readable storage medium is also provided. The storage medium stores a computer program / instruction, and the computer program / instruction implements the above method when executed by the processor. The storage medium may include, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0088] A person skilled in the art can easily understand the implementation structure, working principle and beneficial effects of the mobile terminal-based conference speech management system, electronic device and computer-readable storage medium by reading the above-mentioned mobile terminal-based conference speech management method. For the sake of brevity, it will not be repeated here.

[0089] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0090] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0092] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0093] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various application aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the application point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0094] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0095] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0096] The various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some modules in the electronic device according to the embodiment of the present application. The present application may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0097] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0098] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A conference speech management method based on a mobile terminal, characterized in that: Applied to a conference speech management system, the conference speech management system comprises a mobile terminal, a server and a speaker, and the mobile terminal has a microphone; The method comprises: establishing a communication channel with the mobile terminal at least upon receiving access request information sent by the mobile terminal; When receiving a first speaking instruction sent by the mobile terminal, acquiring initial audio information input through a microphone of the mobile terminal; The speaker is used to play the user audio information in the initial audio information.

2. The method according to claim 1, characterized in that The access request information is generated by the mobile terminal by executing the following steps: Accessing the access interface of the server according to the service address of the server; The access request information is generated according to the identity information input by the user in the access interface.

3. The method according to claim 2, characterized in that Before executing the step of accessing the access interface of the server according to the service address of the server, the mobile terminal further executes the following steps: Scan the QR code corresponding to the server to obtain the service address.

4. The method according to claim 1, characterized in that: The step of establishing a communication channel with the mobile terminal at least upon receiving the access request information sent by the mobile terminal comprises: Upon receiving the access request information, establishing a communication channel with the mobile terminal; Alternatively, the access request information includes identity information of the user, and the establishing of a communication channel with the mobile terminal at least upon receiving the access request information sent by the mobile terminal includes: Upon receiving the access request information, comparing the identity information corresponding to the access request information with each identity information in a preset list, the preset list including at least one identity information; When the identity information corresponding to the access request information is the same as any identity information in the preset list, establishing a communication channel with the mobile terminal; Alternatively, the access request information includes a mobile phone number of the user, and the establishing of a communication channel with the mobile terminal at least upon receiving the access request information sent by the mobile terminal includes: Upon receiving the access request information, sending first verification information to the mobile phone number corresponding to the access request information; When receiving the second verification information sent by the mobile terminal, comparing the second verification information with the first verification information; When the second verification information is the same as the first verification information, a communication channel with the mobile terminal is established.

5. The method according to any one of claims 1 to 4, characterized in that: Before using the speaker to play the user audio information in the initial audio information, the method further includes: performing acoustic echo cancellation on the initial audio information at least according to the echo audio information to obtain user audio information, wherein the echo audio information is determined based on the last audio information played by the speaker; The user audio information is sent to the speaker so as to play the user audio information by using the speaker.

6. The method according to claim 5, characterized in that The performing acoustic echo cancellation on the initial audio information at least according to the echo audio information to obtain the user audio information includes: Based on the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, the feedback compensation coefficient and the echo audio information, the initial audio information is processed to obtain the user audio information.

7. The method according to claim 6, characterized in that The processing of the initial audio information based on the network delay between the mobile terminal and the server, the spatial transmission delay between the mobile terminal and the server, the feedback compensation coefficient, and the echo audio information to obtain the user audio information includes: processing the initial audio information by the following formula: e(n)=(T now -T ntl -T stl )×d(n)-cf×y(n) Where, e(n) represents the user audio information, T now Indicates the current server time, T stl represents the spatial transmission delay, T ntl represents the network delay, d(n) represents the initial audio information, cf represents the feedback compensation coefficient, and y(n) represents the echo audio information.

8. The method according to claim 6, characterized in that After establishing the communication channel with the mobile terminal, the method further includes: Receiving test information sent by the mobile terminal, the test information including first time information, the first time information being the time when the mobile terminal sends the test information; When receiving the test information, sending feedback information to the mobile terminal; Acquire second time information from the mobile terminal, where the second time information is the time when the mobile terminal receives the feedback information; Determining the network delay based on a difference between the second time information and the first time information; and / or, After establishing the communication channel with the mobile terminal, the method further includes: Playing preset audio information using the speaker; Receiving feedback audio information collected by the mobile terminal, where the feedback audio information is obtained by collecting the preset audio information using a microphone of the mobile terminal; Determining the spatial transmission delay based on a difference between the fourth time information and the third time information; The third time information is the time when the preset audio information is sent, and the fourth time information is the time when the feedback audio information is received.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images