Image control method, image control device, and program

Abstract

Provided is an image control method capable of clearly transmitting, to conference participants, the meanings of not speaking or wanting to speak. The image control method determines a mute operation, transmits control information to a camera based on the determination result, the control information switching between a first state and a second state, and outputs first image information in the first state and second image information in the second state. And outputs the first image information or the second image information on the basis of the control information.

Description

Technical Field

[0001] One embodiment of the present invention relates to an image control method, an image control device, and a program. Background Art

[0002] Patent Document 1 discloses an information processing method that, when it is determined that a user is in a speaking state, outputs fast-forwarded image data of a user obtained by fast-forwarding image data from a user a predetermined time ago among cached image data to other users.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2022-16997 Summary of the invention

[0004] With the information processing method of Patent Document 1, since the entire screen display does not change regardless of whether the microphone is muted on or off, it is difficult to convey to other users that the user intends not to speak or wants to speak.

[0005] Therefore, an object of one embodiment of the present invention is to provide an image control method capable of clearly conveying to conference participants the intention of not speaking or wanting to speak.

[0006] An image control method determines a mute operation, and based on the determination result, sends control information to a camera, wherein the control information switches a first state and a second state, and for a camera that outputs first image information in the first state and outputs second image information in the second state, the first image information or the second image information is output based on the control information.

[0007] Effects of the Invention

[0008] According to one embodiment of the present invention, it is possible to clearly convey to conference participants the intention of not speaking or wanting to speak. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic elevation diagram of a room where an image control system is installed.

[0010] Figure 2 It is a block diagram showing the structure of an image control system.

[0011] Figure 3 This is a flowchart showing the operation of the image control system.

[0012] Figure 4 This is a diagram showing an example of the first image information P1 output in the first state.

[0013] Figure 5This is a diagram showing an example of the second image information P2 output in the second state.

[0014] Figure 6 This is a block diagram showing the structure of an image control system according to Modification 1.

[0015] Figure 7 This is a flowchart showing the operation of the image control system according to Modification 1.

[0016] Figure 8 This is a flowchart showing the operation of the image control system according to Modification Example 2.

[0017] Fig. 9 This is a block diagram showing the structure of an image control system according to Modification 3.

[0018] Fig.10 This is a flowchart showing the operation of the image control system according to Modification 3.

[0019] Fig.11 This is a block diagram showing the structure of an image control system according to Modification 4.

[0020] Fig.12 This is a flowchart showing the operation of the image control system according to Modification 4. DETAILED DESCRIPTION

[0021] Figure 1 It is a schematic elevation diagram of a room where an image control system is installed. Figure 2 2 is a block diagram showing the structure of an image control system. The image control system includes a microphone 10, a controller 20, a camera 30, and a personal computer (PC) 40. The microphone 10, the controller 20, the camera 30, and the PC 40 are connected via a network.

[0022] The microphone 10 is installed on the ceiling of the room. The microphone 10 has a thin rectangular parallelepiped frame. The controller 20 and the camera 30 are installed on the table.

[0023] A table is provided directly below the housing of the microphone 10. Figure 1 In the example, there are multiple users (users u1, u2) around the table.

[0024] The camera 30 acquires an image of the user, performs predetermined signal processing on an image signal related to the acquired image, and transmits the image signal after the signal processing to the PC 40. The camera 30 performs framing processing by, for example, panning, tilting, or zooming.

[0025] The microphone 10 acquires the voice of the user and includes a communication I / F 11 , a control unit 12 , a flash memory 13 , a RAM 14 , and a microphone unit 15 .

[0026] The control unit 12 reads the operation program from the flash memory 13 to the RAM 14, thereby comprehensively controlling the operation of the microphone 10. The program does not need to be stored in advance in the flash memory 13 of the device. The control unit 12 may download the program from a server or the like and read it to the RAM 14 each time, for example.

[0027] The control unit 12 functions as a processor for processing an audio signal. The control unit 12 performs predetermined signal processing on the audio signal obtained by the microphone unit 15. The microphone unit 15 is, for example, an array microphone having a plurality of microphone units. The control unit 12 performs directivity processing of beamforming. Beamforming is a process of forming a sound pickup beam with improved sensitivity in the direction of the speaker by, for example, delaying and processing so that the phase is aligned in the direction of the speaker.

[0028] The control unit 12 can obtain the direction information of the speaker's voice and make the sound pickup beam face the speaker's direction. The control unit 12 analyzes the sound signals obtained from the multiple microphones in the microphone unit 15 to estimate the arrival direction of the voice. The sound signal analysis method can be any method such as the cross-correlation method, the delay-and-sum method, or the MUSIC (Multiple Signal Classification) method. In the cross-correlation method, the control unit 12 calculates the cross-correlation of the sound signals of the multiple microphones, for example. The control unit 12 obtains the peak value of the cross-correlation of the sound signals of two microphones, for example. Furthermore, the control unit 12 obtains the peak value of the cross-correlation of the sound signals of the other two microphones. The control unit 12 estimates the arrival direction of the voice based on the multiple cross-correlation peaks calculated in this way. That is, the control unit 12 selects more than two groups of multiple microphones to obtain multiple cross-correlation peaks. The estimated arrival direction of the voice is represented by a space vector, for example.

[0029] The control unit 12 sends the processed audio signal to the PC 40 via the communication I / F 11. The PC 40 is connected to other information processing devices at a remote location via a network such as the Internet. The PC 40 sends the audio signal received from the microphone 10 and the image signal received from the camera 30 to the information processing device at the far end. The PC 40 can display the image signal received from the camera 30 on a display (not shown) of the device.

[0030] In addition, the PC 40 receives an image signal and an audio signal from an information processing device at a remote end. The PC 40 outputs the received image signal to a display (not shown). In addition, the PC 40 outputs the received audio signal to a speaker (not shown). Thus, the image control system functions as a component of a teleconference system for conducting a conference with a remote location.

[0031] The controller 20 is an example of the image control device of the present invention, and is a remote controller for operating the microphone 10 or the camera 30. The controller 20 includes a communication I / F 21, a control unit 22, a flash memory 23, a RAM 24, and a user I / F 25.

[0032] The control unit 22 reads the operation program 231 from the flash memory 23 to the RAM 24, thereby comprehensively controlling the operation of the controller 20. The program does not need to be stored in advance in the flash memory 23 of the device. The control unit 22 may download the program from a server or the like and read it to the RAM 24, for example.

[0033] The control unit 22 receives an operation from a user via the user I / F 25. The user I / F 25 has at least a mute button. However, the user I / F 25 may also have other operating elements such as a volume change button and a power button.

[0034] Figure 3 2 is a flowchart showing the operation of the image control system. The control unit 22 of the controller 20 first determines the mute operation (S11). The control unit 22 sends a mute message to the microphone 10 based on the determination result (S12). The mute operation is received via the user I / F 25. The mute operation includes a mute on operation and a mute off operation. When the control unit 22 determines that the mute on operation is performed, the mute on message is sent to the microphone 10 as the mute message. When the control unit 22 determines that the mute off operation is performed, the mute off message is sent to the microphone 10 as the mute message.

[0035] The microphone 10 receives the mute information (S21). When receiving the mute on information, the microphone 10 stops outputting the sound signal obtained by the microphone unit 15 and changes to the mute state. When receiving the mute off information, the microphone 10 resumes outputting the sound signal obtained by the microphone unit 15 and cancels the mute state.

[0036] The control unit 22 of the controller 20 sends control information for switching between the first state and the second state to the camera 30 based on the determination result (S13). The control unit 22 sends control information for switching to the first state to the camera 30 when determining that the mute on operation is performed, and sends control information for switching to the second state to the camera 30 when determining that the mute off operation is performed.

[0037] The camera 30 receives the control information (S31). The camera 30 switches the camera state based on the control information (S32). The camera state includes a first state for outputting the first image information and a second state for outputting the second image information.

[0038] Figure 4 FIG. 1 is a diagram showing an example of first image information P1 output in the first state. The first state corresponds to a predetermined reference state. The reference state is, for example, a state of full-body shooting in which all of a plurality of users are shot. Figure 4 In the example of , the framing process is performed so that the first image information P1 includes images of both the user u1 and the user u2.

[0039] Alternatively, the reference state may correspond to a state in which the image of a specific user is focused. The specific user is a presenter who is in charge of the conference. The specific user is preset by the PC 40. Alternatively, the reference state is an initial state in which framing processing such as panning, tilting, and zooming is not performed.

[0040] The second state is a state in which the speaker is focused by performing framing processing such as panning, tilting, and zooming. As an example, the camera 30 performs processing for recognizing the face of the speaker by using a predetermined model such as a neural network. The camera 30 performs panning, tilting, and zooming so that the image of the recognized speaker becomes the center of the screen and the occupancy rate of the image of the speaker in the screen becomes a predetermined ratio (e.g., 50%).

[0041] Figure 5 FIG. 1 is a diagram showing an example of the second image information P2 output in the second state. Figure 5 In the example of FIG. 1 , the camera 30 recognizes the user u1 as the speaker. The second image information P2 is framed so that the image of the user u1 occupies 50% of the screen.

[0042] The camera 30 may receive the direction information of the speaker's voice obtained by the microphone 10 and perform framing processing based on the direction information. In addition, the camera 30 may perform processing to block images of persons other than the speaker or persons not attending the meeting in the second state.

[0043] As described above, the image control system of this embodiment changes the camera state of the camera 30 in conjunction with the mute button of the controller 20. When the user performs the mute on operation, the first image information output by the camera 30 becomes an image of the entire user. When the user performs the mute off operation, the second image information output by the camera 30 becomes an image focusing on the speaker (in Figure 5 In the example, the image of user u1) is used.

[0044] The first image information and the second image information are sent to the remote information processing device via the PC 40. The first image information or the second image information is displayed on the display of the remote information processing device. Alternatively, the PC 40 may display the first image information and the second image information on a display of the device (not shown).

[0045] Typically, the mute-off state and the mute-on state are displayed on the GUI of the software of the remote conference system used to conduct a conference with a remote location. Therefore, as a comparison, when, for example, the camera state does not change even if the user performs a mute operation, it is difficult for the user of the information processing device on the remote side to notice the change from mute-on to mute-off, or from mute-off to mute-on.

[0046] However, in the image control system of the present embodiment, when the user performs the mute on operation, the image of the camera 30 becomes an image that captures all users. The user who observes the image that captures all users can intuitively understand that there is no one who wants to speak. In the image control system of the present embodiment, when the user performs the mute off operation, the image of the camera 30 becomes an image that focuses on the speaker (in the Figure 5 In the example of FIG. 1 , the image of the user u1 is enlarged. In addition, the user who observes the enlarged image of the user u1 can intuitively understand that the user u1 intends to speak.

[0047] As described above, the user of the image control system of the present embodiment can obtain a new customer experience in which the user can clearly convey to other conference participants the intention of not speaking or wanting to speak.

[0048] (Variant 1)

[0049] Figure 6 FIG. 1 is a block diagram showing the structure of an image control system according to Modification 1. Figure 2 The same reference numerals are used to denote common structures, and description thereof will be omitted. Figure 7 FIG. 1 is a flowchart showing the operation of the image control system according to Modification 1. Figure 3 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0050] In the image control system according to Modification 1, the camera 30 is directly connected to the PC 40. In the process of S13, the controller 20 transmits control information to the PC 40, and the control information switches between the first state and the second state.

[0051] The PC 40 receives the control information (S41). Based on the received control information, the PC 40 switches the state of the camera 30 (S42). As described above, the camera state includes the first state of outputting the first image information and the second state of outputting the second image information.

[0052] As described above, the controller 20 may send control information to the camera 30 via an information processing device that receives an image signal from the camera.

[0053] (Variant 2)

[0054] Figure 8 FIG. 1 is a flowchart showing the operation of the image control system according to Modification 2. Figure 3 The same reference numerals are used to designate the common configurations, and description thereof will be omitted. The controller 20 according to the second modification transmits control information to the camera 30 via the microphone 10 .

[0055] Specifically, in the image control system according to Modification 2, when the controller 20 transmits the mute information to the microphone 10 in the process of S12, the microphone 10 transmits the control information for switching between the first state and the second state to the camera 30 (S23).

[0056] That is, in the second modification, the controller 20 does not transmit the control information to the camera 30 but transmits the mute information to the microphone 10 , whereby the microphone 10 transmits the control information to the camera 30 .

[0057] Furthermore, when the camera 30 receives the direction information of the speaker's voice obtained by the microphone 10 and performs framing processing based on the direction information, the microphone 10 may stop sending the direction information when sending the control information for switching to the first state to the camera 30. Thus, when the camera 30 switches to the first state, it is possible to prevent the conflict between the framing processing for focusing on the speaker based on the direction information and the framing processing for photographing all of the plurality of users.

[0058] (Variant 3)

[0059] Fig. 9 FIG. 4 is a block diagram showing the structure of an image control system according to Modification 3. Figure 2 The same reference numerals are used to denote common structures, and description thereof will be omitted. Fig.10 FIG. 1 is a flowchart showing the operation of the image control system according to Modification 3. Figure 3 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0060] The image control system according to the third modification includes a processor 50. The processor 50 is hardware for signal processing (DSP: Digital Signal Processor). The processor 50 controls devices such as the microphone 10, a speaker (not shown), or the camera 30 connected to the image control system via a network, and performs signal processing such as routing, mixing, or effects on signals input to or output from each device.

[0061] The processor 50 may receive the sound signal obtained by the microphone unit 15 from the microphone 10 and perform beamforming directivity processing. Alternatively, the processor 50 may analyze the sound signals obtained from the multiple microphones of the microphone unit 15 to estimate the direction of arrival of the voice. Alternatively, the processor 50 may also receive the image signal captured by the camera 30 and perform framing processing.

[0062] The controller 20 is directly connected to the processor 50. In the process of S12, the controller 20 sends the mute information to the processor 50. The processor 50 receives the mute information (S51) and sends the mute information to the microphone 10 (S52). Then, the processor 50 sends control information for switching between the first state and the second state to the camera 30 (S53).

[0063] As described above, in the case where the image control system has the processor 50 , the controller 20 may transmit control information to the camera 30 via the processor 50 .

[0064] (Variant 4)

[0065] Fig.11 FIG. 4 is a block diagram showing the structure of an image control system according to Modification Example 4. Figure 6 The same reference numerals are used to denote common structures, and description thereof will be omitted. Fig.12 FIG. 4 is a flowchart showing the operation of the image control system according to Modification 4. Figure 7 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0066] In the image control system according to Modification 4, the camera 30 and the controller 20 are directly connected to the PC 40. The controller 20 transmits mute information to the PC 40 in the process of S12.

[0067] When receiving the mute information (S61), the PC 40 transmits the mute information to the microphone 10 (S62). Then, the PC 40 transmits control information for switching between the first state and the second state to the camera 30 (S63).

[0068] As described above, the controller 20 may send the mute information to the microphone 10 via the PC 40 and send the control information to the camera 30 .

[0069] It should be noted that the description of the present embodiment is illustrative in all aspects and is not restrictive. The scope of the present invention is not shown by the above-mentioned embodiment, but by the claims. Moreover, the scope of the present invention includes the scope equivalent to the claims.

[0070] For example, the devices including the processor 50 do not need to be connected via a network, but may be connected via other communication lines such as USB. Alternatively, the devices including the processor 50 may be connected via a wireless method such as a wireless LAN or Bluetooth (registered trademark).

[0071] Description of the label

[0072] 10: Microphone, 11: Communication I / F, 12: Control unit, 13: Flash memory, 14: RAM, 15: Microphone unit, 20: Controller, 21: Communication I / F, 22: Control unit, 23: Flash memory, 24: RAM, 25: User I / F, 30: Camera, 50: Processor

Claims

1. An image control method, wherein: Determine the silent operation, Based on the determination result, control information is sent to the camera, and the control information switches between the first state and the second state. The camera that outputs the first image information in the first state and outputs the second image information in the second state outputs the first image information or the second image information based on the control information.

2. The image control method according to claim 1, wherein: The mute operation includes a mute on operation and a mute off operation. When the mute on operation is determined, control information for switching to the first state is sent to the camera, When the mute off operation is determined, control information for switching to the second state is sent to the camera, The first state corresponds to a predetermined reference state. The first image information includes an image corresponding to the reference state.

3. The image control method according to claim 2, wherein: The reference state corresponds to a state in which all of the plurality of users are photographed or an image of a specific user is focused.

4. The image control method according to claim 3, wherein: The second state corresponds to a state in which the image of the speaker is focused. The second image information includes an image in which the speaker is focused.

5. The image control method according to any one of claims 1 to 4, wherein: The control information is sent to the camera via a processor that receives an image signal from the camera.

6. The image control method according to any one of claims 1 to 4, wherein: The control information is sent to the camera via an information processing device that receives an image signal from the camera.

7. The image control method according to any one of claims 1 to 4, wherein: Based on the determination result, muting information is transmitted to the microphone.

8. The image control method according to claim 7, wherein: The control information is sent to the camera via the microphone.

9. The image control method according to claim 7, wherein: The microphone obtains the direction information of the speaker's voice and sends the direction information to the camera. The camera focuses on the image of the speaker based on the direction information.

10. The image control method according to any one of claims 1 to 4, wherein: The acquired first image information or the acquired second image information is displayed on a display.

11. An image control device comprising a control unit, The control unit determines the silent operation, Based on the determination result, control information is sent to the camera, and the control information switches between the first state and the second state. The camera that outputs the first image information in the first state and outputs the second image information in the second state outputs the first image information or the second image information based on the control information.

12. The image control device according to claim 11, wherein: The mute operation includes a mute on operation and a mute off operation. When the control unit determines that the mute on operation has been performed, the control unit sends control information for switching to the first state to the camera. When the mute off operation is determined, control information for switching to the second state is sent to the camera, The first state corresponds to a predetermined reference state. The first image information includes an image corresponding to the reference state.

13. The image control device according to claim 12, wherein: The reference state corresponds to a state in which all of the plurality of users are photographed or an image of a specific user is focused.

14. The image control device according to claim 13, wherein: The second state corresponds to a state in which the image of the speaker is focused. The second image information includes an image in which the speaker is focused.

15. The image control device according to any one of claims 11 to 14, wherein: The control unit transmits the control information to the camera via a processor that receives an image signal from the camera.

16. The image control device according to any one of claims 11 to 14, wherein: The control unit transmits the control information to the camera via an information processing device that receives an image signal from the camera.

17. The image control device according to any one of claims 11 to 14, wherein: The control unit transmits muting information to the microphone based on the determination result.

18. The image control device according to claim 17, wherein: The control unit transmits the control information to the camera via the microphone.

19. The image control device according to claim 17, wherein: The microphone obtains the direction information of the speaker's voice and sends the direction information to the camera. The camera focuses on an image of the speaker based on the direction information.

20. The image control device according to any one of claims 11 to 14, wherein: The control unit displays the acquired first image information or the second image information on a display.

21. A program that causes an information processing device to execute the following processing: Determine the silent operation, Based on the determination result, control information is sent to the camera, and the control information switches between the first state and the second state. The camera that outputs the first image information in the first state and outputs the second image information in the second state outputs the first image information or the second image information based on the control information.

Images