Audio processing method and audio processing device

Abstract

To provide an audio processing method capable of preventing audio from being not transmitted due to unintentional mute-on or audio from being transmitted unintentionally due to mute-off according to a conference format.SOLUTION: An audio processing method includes conference formats categorized into a general conference in which audio is transmitted from a communication terminal of a predetermined transmission source to a plurality of communication terminals, and an individual conference in which audio is transmitted from the communication terminal of the predetermined transmission source to a communication terminal selected from the plurality of communication terminals, and controls the mute state of the audio transmitted from the communication terminal of the predetermined transmission source according to the conference formats.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an audio processing method and an audio processing device. [Background technology]

[0002] Patent Document 1 discloses an invention for adjusting the volume of a general conference and an individual conference that are being participated in at the same time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-170803 Summary of the Invention [Problem to be solved by the invention]

[0004] The invention of Patent Document 1 is not an invention for adjusting the volume when changing the conference format.

[0005] One of the present embodiments aims to provide an audio processing method that can prevent unintentional muting, which results in audio not being transmitted, or unintentional muting, which results in audio being transmitted, depending on the conference format. [Means for solving the problem]

[0006] The audio processing method includes conference formats categorized into a general conference in which audio is transmitted from a communication terminal of a specified source to multiple communication terminals, and an individual conference in which audio is transmitted from the communication terminal of the specified source to a communication terminal selected from the multiple communication terminals, and controls the mute state of the audio transmitted from the communication terminal of the specified source according to the conference format. [Effects of the Invention]

[0007] According to one embodiment of the present invention, it is possible to prevent unintentional muting, which prevents audio from being transmitted, or unintentional muting, which prevents audio from being transmitted, depending on the conference format. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing the configuration of an audio conference system 1. FIG. [Figure 2] This is a schematic elevation view of the interior. [Figure 3] FIG. 2 is a block diagram showing the configuration of a signal processing device 20. [Figure 4] FIG. 2 is a block diagram showing the configuration of a user terminal 40. [Figure 5] FIG. 2 is a diagram showing an example of a video conference screen displayed on the display device 10. [Figure 6] FIG. 2 is a diagram showing an example of a video conference screen displayed on the display device 10. [Figure 7] 4 is a flowchart showing the operation of the processor 14. [Figure 8] 10 is a flowchart showing the operation of a processor 14 according to a first modification. [Figure 9] 10 is a flowchart showing the operation of the processor 14 according to the second modification. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1 is a block diagram showing the configuration of an audio conference system 1 of this embodiment. The audio conference system 1 includes a display device 10, a signal processing device 20, a personal computer (PC) 30, a user terminal 40, and a user terminal 41. The signal processing device 20 corresponds to the audio processing device of the present invention and is connected to the display device 10, the PC 30, the user terminal 40, and the user terminal 41. The signal processing device 20 is connected to a PC in a remote location via a network. Note that the user terminal 40, the user terminal 41, and the PC 30 may be connected to the signal processing device 20 via a network.

[0010] FIG. 2 is a schematic elevation view of a room. As an example, the display device 10 is installed on a wall in the room. As an example, the signal processing device 20 is attached to the ceiling. There are multiple users (users u1 and u2) around a desk. A user terminal 41 is installed on the desk in front of user u1. A user terminal 40 is installed on the desk in front of user u2.

[0011] 3 is a block diagram showing the configuration of the signal processing device 20. The signal processing device 20 includes a camera 11, a speaker 12, a plurality of microphones 13, a processor 14, a memory 15, and an interface (I / F) 16.

[0012] The memory 15 is a storage medium that stores an operation program for the processor 14. The processor 14 reads the operation program from the memory 15 and performs various operations. The program does not have to be stored in the memory 15. For example, the program may be stored in a storage medium of an external device such as a server. In this case, the processor 14 simply reads the program from the server and executes it each time.

[0013] The processor 14 receives sound signals acquired by the multiple microphones 13. The processor 14 performs predetermined sound processing on the sound signals acquired by the multiple microphones 13. For example, the processor 14 performs beamforming on the sound signals acquired by the multiple microphones 13. Beamforming is a process of forming a sound collection beam with directionality in a predetermined direction by adding delays to the sound signals acquired by the multiple microphones 13 and combining them. The sound collection beam can also be formed with directionality that focuses on a predetermined position.

[0014] The processor 14 sets the sound collection beam of the microphone 13 in the direction of the user terminals 40, 41 based on the position information of the user terminals 40, 41. This allows the sound collection beam to be directed toward users who are near the user terminals 40, 41. The position information of the user terminals 40, 41 is stored in advance in the memory 15, for example. The users input the position information of the user terminals 40, 41 in advance. The signal processing device 20 receives the position information of the user terminals 40, 41 via a GUI executed on the PC 30, for example, as an application program of the PC 30. The positions of the user terminals 40, 41 may also be determined based on images acquired by the camera 11, for example. The processor 14 recognizes the user terminals 40, 41 from images from the camera 11 using a trained algorithm that has been trained to recognize the user terminals 40, 41 from images from the camera 11 using a predetermined algorithm, such as a neural network. The processor 14 determines the position information of the user terminals 40, 41 using control information from the camera 11 that has recognized the user terminals 40, 41. Alternatively, the processor 14 may obtain the position information of the user terminals 40, 41 by referring to a table or function that defines the relationship between the images of the user terminals 40, 41 on the image acquired by the camera 11 and the position information of the user terminals 40, 41. If the microphone 13 and the camera 11 are in different positions, the processor 14 also obtains relative position information of the microphone 13 and the camera 11, and obtains the position information of the user terminal 40 based on the positional relationship between the microphone 13 and the user terminals 40, 41.

[0015] The processor 14 may form a sound collection beam that focuses on the position of the speaker. The processor 14 detects the position of the speaker by, for example, calculating a correlation value between sound signals from a plurality of microphones 13 to determine the difference (phase difference) in the timing at which the voices are acquired. The processor 14 can uniquely determine the position of the speaker by calculating the difference in the timing at which the voices are acquired from three or more microphones 13.

[0016] It should be noted that at least two or more microphones 13 can form a sound collection beam.

[0017] The processor 14 outputs a sound signal related to the sound collection beam to the I / F 16. The I / F 16 is a communication I / F such as LAN, USB, HDMI (registered trademark), or Bluetooth (registered trademark). The I / F 16 is connected to the PC 30, for example, via USB. The I / F 16 is also connected to the display device 10, for example, via HDMI (registered trademark). The I / F 16 is also connected to the user terminals 40 and 41 via Bluetooth (registered trademark). The I / F 16 is also connected to a network via a LAN terminal.

[0018] FIG. 4 is a block diagram of user terminal 40. Because user terminal 41 and user terminal 40 have the same configuration, FIG. 4 shows the configuration of user terminal 40 as a representative. User terminal 40 is an example of a communication terminal of the present invention. User terminal 40 includes a communication I / F 21, a control unit 22, a flash memory 23, a RAM 24, a user I / F 25, and an LED 26, which is an example of a display. Control unit 22 comprehensively controls the operation of user terminal 40 by reading an operating program from flash memory 23 to RAM 24. Note that the program does not need to be stored in the flash memory 23 of the device itself. Control unit 22 may download the program from a server or the like and read it into RAM 24 whenever necessary. Control unit 22 accepts user operations via user I / F 25.

[0019] The user I / F 25 has, for example, an operator for accepting an operation to change the volume of the speaker 12, an operator for accepting an operation to change the framing of the camera 11, or an operator for accepting an operation to mute on / off the microphone. These operators may be physically separate operators, or may be the same operator. For example, the user terminal 40 accepts a mute-on operation when it accepts a single press of the operator in a short period of time, and accepts a mute-off operation when it accepts a continuous long press of the operator for a predetermined period of time or more. Note that the operation of the user terminal 40 may be executed on the PC 30 as an application program of the PC 30.

[0020] The signal processing device 20 transmits a sound signal relating to the sound collection beam to a PC at a remote location via a network.

[0021] The signal processing device 20 receives an audio signal from a remote PC via a network. The processor 14 outputs the audio signal received via the network to the speaker 12. The speaker 12 emits the audio signal received from the processor 14.

[0022] This allows the user of the signal processing device 20 to hold an audio conference with a user in a remote location.

[0023] Processor 14 also receives a video signal related to the video captured by camera 11. Processor 14 performs predetermined signal processing on the video signal captured by camera 11. The signal processing is, for example, framing processing using pan, tilt, or zoom. Processor 14 transmits the video signal after sound signal processing to a PC at a remote location via a network.

[0024] The signal processing device 20 receives a video signal from a remote PC via a network. The signal processing device 20 outputs the received video signal to the display device 10. The display device 10 receives an input of the video signal from the signal processing device 20. The display device 10 displays an image based on the received video signal.

[0025] This allows the user of the signal processing device 20 to hold a video conference with a user in a remote location.

[0026] It should be noted that the camera 11 and the speaker 12 are not essential to the present invention. It is also not essential to install the camera 11, the speaker 12, and the microphone 13 on the ceiling. For example, in the present invention, the microphone 13 may be installed on the ceiling, and the camera 11 and the speaker 12 may be set on a desk.

[0027] Next, Fig. 5 is a diagram showing an example of a video conference screen displayed on the display device 10. The example in Fig. 5 shows a video conference at four locations (locations P1, P2, P3, and P4). The example in Fig. 5 shows a general conference in which audio and video are transmitted from a PC at each location to PCs at all other locations. This type of general conference is used in a format in which, for example, a participant at one location actively speaks, while participants at other locations listen and listen. In other words, participants at other locations often do not speak during a general conference.

[0028] FIG. 6 is a diagram showing an example of a video conference screen displayed on the display device 10. The example in FIG. 6 shows an individual conference in which audio and video are transmitted from communication terminals at various locations to a selected communication terminal. For example, a PC at location P1 transmits audio and video to a PC at location P2. The PC at location P2 transmits audio and video to a PC at location P1. The PC at location P3 transmits audio and video to a PC at location P4. The PC at location P4 transmits audio and video to a PC at location P3. Such individual conferences are used, for example, in a format for two-way conversation between specific multiple locations. In other words, in individual conferences, all participants often speak.

[0029] In this way, the audio conference system of this embodiment has conference formats classified into general conferences and individual conferences.

[0030] 7 is a flowchart showing the operation of the processor 14. The processor 14 controls the mute state of the audio transmitted from a communication terminal of a predetermined transmission source in accordance with the conference format.

[0031] First, the processor 14 determines the type of the conference (S11). For example, the processor 14 receives information about the type of the conference from the PC 30. A videoconferencing application program executed on the PC 30 transmits information about the type of the current conference to the processor 14 via an API (Application Programming Interface). Alternatively, the processor 14 may obtain information about the type of the current conference by analyzing the video signal received via the I / F 16. For example, as shown in FIG. 5, the processor 14 determines that the type of the current conference is a general conference when video of participants at all locations is included.

[0032] When the processor 14 determines that the event is a general conference, it controls the mute state of the audio transmitted from the signal processing device 20 to a mute-on state (S12). The processor 14 sets the mute-on state, for example, by not outputting the audio signal received from the microphone 13 to the PC 30. Alternatively, the processor 14 sets the mute-on state, for example, by setting the microphone 13 so that the sound collection beam is not directed toward the user terminals 40 and 41. In other words, controlling the mute state includes controlling the sound collection beam of the microphone 13.

[0033] If the processor 14 determines that the conference is an individual conference, it controls the mute state of the audio transmitted from the signal processing device 20 to a mute-off state (S13). The processor 14 controls the mute-off state by, for example, setting the audio signal received from the microphone 13 to a state in which it is output to the PC 30. Alternatively, the processor 14 sets the mute-off state by, for example, setting the microphone 13 to direct a sound collection beam in the direction of the user terminals 40 and 41.

[0034] As described above, in general conferences, participants at certain locations often speak proactively, while participants at other locations do not. In general conferences, participants at other locations may not intend to speak, but if their mutes are off, their voices may be transmitted unintentionally. Furthermore, in individual conferences, all participants often speak. In individual conferences, all participants often intend to speak, but if their mutes are unintentionally on, their voices may not be transmitted even though they are speaking. In response to this, the processor 14 of this embodiment controls the mute to on during general conferences and to off during individual conferences. Therefore, users of the voice processing method of this embodiment can enjoy a new customer experience, preventing unintentional mute-on, which prevents voices from being transmitted, or unintentional mute-off, which prevents voices from being transmitted unintentionally, depending on the conference format. Note that controlling the mute to on during general conferences and the mute to off during individual conferences is just one example. Depending on the conference participants' intention to speak, the mute may be controlled to off during general conferences or to on during individual conferences.

[0035] (Variation 1) 8 is a flowchart showing the operation of the processor 14 according to the first modification. The processor 14 according to the first modification first accepts a change in the conference format (S10). A change in the conference format is a change from a general conference to an individual conference, or a change from an individual conference to a general conference. When the processor 14 accepts a change in the conference format, it performs the processes from S11 onwards and controls the mute state according to the conference format. Note that controlling the mute to the on state in the general conference and the mute to the off state in the individual conference is just one example, and the mute may be controlled to the off state in the general conference or the mute to the on state in the individual conference depending on the intention of the conference participants to speak.

[0036] This allows the processor 14 to control the mute state to be appropriate for the changed conference format in the initial state after the change of the conference format.

[0037] (Variation 2) 9 is a flowchart showing the operation of the processor 14 according to the modification 2. The processor 14 according to the modification 2 accepts a change in the conference format (S21). When accepting the change in the conference format, the processor 14 controls the mute state of the audio transmitted from the signal processing device 20 to a mute-on state (S22).

[0038] That is, the processor 14 controls the mute to be on when it accepts a change in conference format from an individual conference to a general conference, or when it accepts a change in conference format from a general conference to an individual conference. The processor 14 always sets the mute to be on as the initial state after the conference format is changed.

[0039] A user of the audio processing method of the second modification can prevent audio from being transmitted unintentionally due to muting being turned off when the conference format is changed.

[0040] (Variation 3) The processor 14 of the third modification changes the display mode of the LED 26 of the user terminals 40 and 41 depending on the audio mute state. For example, when the mute is off, the processor 14 lights up the LED 26 of the user terminals 40 and 41 in green. For example, when the mute is on, the processor 14 lights up the LED 26 of the user terminals 40 and 41 in red.

[0041] This allows the user to easily understand the current mute status.

[0042] (Variation 4) In the fourth modification, when a change in the conference format is received, the processor 14 notifies the user of the change. The processor 14 announces the change in the conference format by voice, for example, via the speaker 12. After announcing the change in the conference format by voice, the processor 14 controls the mute state of the voice in accordance with the conference format.

[0043] This allows users to know the change in the conference format.

[0044] After providing audio guidance for changing the conference format, processor 14 may wait for the conference format to be changed and accept an operation from the user. After accepting an operation from the user via user terminal 40, 41, processor 14 controls the audio mute state according to the conference format.

[0045] This allows the user to decide to change the conference format at their own will.

[0046] (Variation 5) When a change in the conference format is accepted, the processor 14 according to the fifth modification controls the audio transmitted from a specific communication terminal among the plurality of communication terminals to be muted. The specific communication terminal is, for example, a user terminal used by the conference moderator. The specific communication terminal is, for example, registered in advance in the memory 15. The user registers the user terminal as a specific communication terminal in advance. The signal processing device 20 accepts the registration of the specific communication terminal via a GUI executed on the PC 30, for example, as an application program of the PC 30.

[0047] 2, for example, user u1 is the moderator, and user terminal 41 is the user terminal used by user u1 who is the moderator. User u1 registers user terminal 41 in advance as a specific communication terminal for the moderator.

[0048] For example, when the processor 14 determines that the event is a general conference, it controls the mute state of the audio transmitted from the PC 30 to mute on, but at this time, for example, it sets the microphone 13 so that the sound collection beam is not directed toward the user terminal 40, but is directed toward the user terminal 41. As a result, the user terminal 40 is in mute on state and the user terminal 41 is in mute off state.

[0049] As a result, even if all other communication terminals are in mute-on state, a specific communication terminal such as that used by the conference moderator will be in mute-off state. Therefore, the processor 14 according to the fifth modification can always transmit the voice of a speaking participant to other communication terminals regardless of the conference format.

[0050] The processor 14 may also acquire specific information for identifying a user terminal. The specific information is information unique to each user terminal, such as the serial number or MAC address of the user terminal. A user may register the specific information in the memory 15 to register the user terminal they use as a specific communication terminal. For example, when the processor 14 determines that a general conference is taking place, it controls the other communication terminals to mute on, but directs a sound collection beam in the direction of the user terminal corresponding to the registered specific information and sets it to mute off. This allows a specific user, such as a moderator, to always be muted off regardless of the conference format.

[0051] The processor 14 may recognize a specific communication terminal using the camera 11. For example, the processor 14 may recognize a podium from an image captured by the camera 11, and recognize a user terminal placed on the podium as a specific communication terminal.

[0052] Processor 14 may also recognize a specific communication terminal based on the voice captured by microphone 13 and the voice of another location received via I / F 16. For example, processor 14 may recognize the voice of each user and recognize the user terminal of the user who speaks most frequently as a specific communication terminal. Processor 14 may also recognize a specific communication terminal by analyzing the voice using a natural language processing model. For example, when processor 14 recognizes a statement such as "This group should speak first at the next general meeting," it recognizes the user terminal included in the corresponding individual meeting as a specific communication terminal.

[0053] As a result, when the individual conference is changed to a general conference, the processor 14 can control the user terminal of the user who is speaking to be muted off and the other user terminals to be muted on.

[0054] (Other examples) In the above embodiment and each modified example, user terminals 40 and 41 were shown as an example of a communication terminal. That is, the user terminals 40 and 41 are multiple communication terminals, each connected to a microphone 13 that captures audio and a processor 14 that controls the microphone 13. The processor 14 accepts a change in the conference format and, upon accepting the change, controls the mute state of the microphone 13, thereby controlling the mute state of the user terminals 40 and 41. However, a communication terminal may also be a terminal having multiple configurations and functions of the user terminals 40 and 41, the microphone 13, and the processor 14. For example, the PC 30 can be configured as a communication terminal having all the configurations and functions of a user terminal, a microphone, and a processor. In this case, for example, a videoconferencing application program executed on the PC 30 controls the mute state of audio to be transmitted to other locations according to the conference format. Alternatively, the PC 30 may have the configuration and functions of the processor 14, and the user terminals 40 and 41 may each have the configuration and functions of the microphone 13. In this case, for example, a video conference application program executed on the PC 30 controls the mute state of the audio transmitted from the user terminals 40 and 41 in accordance with the conference format.

[0055] The description of the present embodiment should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined not by the above-described embodiments but by the claims. Furthermore, the scope of the present invention includes the scope equivalent to the claims. [Explanation of symbols]

[0056] 1: Audio conference system, 10: Display device, 11: Camera, 12: Speaker, 13: Microphone, 14: Processor, 15: Memory, 16: I / F, 20: Signal processing device, 40: User terminal, 41: User terminal

Claims

1. a conference format classified into a general conference in which a predetermined source communication terminal transmits audio to a plurality of communication terminals, and an individual conference in which the predetermined source communication terminal transmits audio to a communication terminal selected from the plurality of communication terminals; controlling a mute state of the audio transmitted from the communication terminal of the predetermined source in accordance with the conference format; Audio processing methods.

2. Accept the change in the conference format; When the change is accepted, a mute state of the audio transmitted from the communication terminal of the predetermined transmission source is controlled in accordance with the conference format.

2. The audio processing method according to claim 1.

3. When the change from the individual conference to the general conference is accepted, the audio transmitted from the communication terminal of the predetermined transmission source is controlled to be in a mute-on state.

3. The audio processing method according to claim 2.

4. When the change from the general conference to the individual conference is accepted, the audio transmitted from the communication terminal of the predetermined transmission source is controlled to be in a mute-on state.

3. The audio processing method according to claim 2.

5. When the change is accepted, audio transmitted from a specific communication terminal among the plurality of communication terminals is controlled to be in a mute-off state.

5. The audio processing method according to claim 2.

6. If the change is accepted, notify the user of the change.

5. The audio processing method according to claim 2.

7. each of the plurality of communication terminals is connected to a microphone that captures audio and a controller that controls the microphone; the controller accepts the change, and when the change is accepted, controls the mute state of the microphone, thereby controlling the mute state of the communication terminal.

5. The audio processing method according to claim 2.

8. The control of the mute state includes control of the sound collection beam of the microphone.

8. The audio processing method according to claim 7.

9. changing a display mode of a display device in accordance with the mute state of each of the plurality of communication terminals; The audio processing method according to any one of claims 1 to 4.

10. a conference format classified into a general conference in which a predetermined source communication terminal transmits audio to a plurality of communication terminals, and an individual conference in which the predetermined source communication terminal transmits audio to a communication terminal selected from the plurality of communication terminals; controlling a mute state of the audio transmitted from the communication terminal of the predetermined source in accordance with the conference format; a processor; Audio processing device.

11. The processor: Accept the change in the conference format; When the change is accepted, a mute state of the audio transmitted from the communication terminal of the predetermined transmission source is controlled in accordance with the conference format. The audio processing device according to claim 10.

12. when the processor receives a change from the individual conference to the general conference, the processor controls the audio transmitted from the communication terminal of the predetermined source to be in a mute-on state. The audio processing device according to claim 11 .

13. when the processor accepts a change from the general conference to the individual conference, the processor controls audio transmitted from the communication terminal of the predetermined source to be muted. The audio processing device according to claim 11 .

14. When the processor receives the change, the processor controls audio transmitted from a specific communication terminal among the plurality of communication terminals to be in a mute-off state. The audio processing device according to any one of claims 11 to 13.

15. When the processor accepts the change, it notifies the user of the change. The audio processing device according to any one of claims 11 to 13.

16. each of the plurality of communication terminals is connected to a microphone that captures audio and a controller that controls the microphone; the controller accepts the change, and when the change is accepted, controls the mute state of the microphone, thereby controlling the mute state of the communication terminal. The audio processing device according to any one of claims 11 to 13.

17. The control of the mute state includes control of the sound collection beam of the microphone.

17. The audio processing device of claim 16.

18. the processor changes a display mode of a display device in accordance with the mute state of each of the plurality of communication terminals. The audio processing device according to any one of claims 10 to 13.

Images