Image control method, image control device, and program

Abstract

To provide an image control method capable of clearly conveying to conference participants the intention of not speaking or wanting to speak.SOLUTION: An image control method determines a mute operation, transmits control information to a camera for switching between a first state and a second state on the basis of the determination result, and causes the camera, which outputs first image information in the first state and second image information in the second state, to output the first image information or the second image information on the basis of the control information.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment of the present invention relates to an image control method, an image control device, and a program. [Background technology]

[0002] Patent document 1 discloses an information processing method in which, when it is determined that a user is in a speaking state, the fast-forwarded image data of the user, which has been fast-forwarded from a predetermined time ago among buffered image data, is output to other users. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-16997 Summary of the Invention [Problem to be solved by the invention]

[0004] In the information processing method of Patent Document 1, the overall screen display does not change whether the microphone is muted or off, making it difficult to convey to other users that one wishes to speak or not.

[0005] Therefore, an object of one embodiment of the present invention is to provide an image control method that can clearly communicate to conference participants the intention of not speaking or wanting to speak. [Means for solving the problem]

[0006] The image control method determines a mute operation, transmits control information to a camera for switching between a first state and a second state based on the determination result, and causes the camera, which outputs first image information in the first state and second image information in the second state, to output the first image information or the second image information based on the control information. Effect of the Invention

[0007] According to one embodiment of the present invention, the intention of not speaking or wanting to speak can be clearly communicated to conference participants. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a schematic elevation view of a room in which the image control system is installed. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an image control system. [Diagram 3] 4 is a flowchart showing the operation of the image control system. [Figure 4] 13 is a diagram showing an example of first image information P1 output in a first state. FIG. [Diagram 5] 13 is a diagram showing an example of second image information P2 output in a second state. FIG. [Figure 6] FIG. 13 is a block diagram showing a configuration of an image control system according to a first modified example. [Figure 7] 13 is a flowchart showing the operation of the image control system according to the first modification. [Figure 8] 13 is a flowchart showing the operation of an image control system according to a second modification. [Figure 9] FIG. 11 is a block diagram showing a configuration of an image control system according to a third modified example. [Figure 10] 13 is a flowchart showing the operation of an image control system according to a third modification. [Figure 11] FIG. 13 is a block diagram showing a configuration of an image control system according to a fourth modified example. [Figure 12] 13 is a flowchart showing the operation of an image control system according to a fourth modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Fig. 1 is a schematic elevation view of a room in which an image control system is installed. Fig. 2 is a block diagram showing the configuration of the image control system. The image control system includes a microphone 10, a controller 20, a camera 30, and a personal computer (PC) 40. The microphone 10, the controller 20, the camera 30, and the PC 40 are connected via a network.

[0010] The microphone 10 is installed on the ceiling of a room. The microphone 10 has a thin rectangular parallelepiped housing. The controller 20 and the camera 30 are installed on a desk.

[0011] A desk is placed directly below the housing of the microphone 10. In the example of Fig. 1, multiple users (users u1, u2) are present around the desk.

[0012] The camera 30 captures an image of the user. It performs predetermined signal processing on a video signal related to the captured image and transmits the processed video signal to the PC 40. The camera 30 performs framing processing, for example, by panning, tilting, or zooming.

[0013] The microphone 10 captures the voice of the user. The microphone 10 includes a communication I / F 11, a control unit 12, a flash memory 13, a RAM 14, and a microphone unit 15.

[0014] The control unit 12 controls the overall operation of the microphone 10 by reading out an operating program from the flash memory 13 to the RAM 14. The program does not need to be stored in the flash memory 13 of the device itself. The control unit 12 may download the program from a server or the like and read it into the RAM 14 each time.

[0015] The control unit 12 functions as a processor that processes sound signals. The control unit 12 performs predetermined signal processing on sound signals acquired by the microphone unit 15. The microphone unit 15 is, for example, an array microphone having a plurality of microphone units. The control unit 12 performs directional processing of beamforming. Beamforming is, for example, a process of forming a sound collection beam with a high sensitivity in the direction of the speaker by aligning the phase in the direction of the speaker using a delay and sum process.

[0016] The control unit 12 may obtain directional information of the speaker's voice and perform processing to direct a sound collection beam in the direction of the speaker. The control unit 12 analyzes sound signals acquired from multiple microphones in the microphone unit 15 to estimate the direction of arrival of the voice. The sound signal analysis method may be any method such as a cross-correlation method, a delay-and-sum method, or a multiple signal classification (MUSIC) method. In the cross-correlation method, the control unit 12 calculates, for example, the cross-correlation of sound signals from multiple microphones. The control unit 12 obtains, for example, a cross-correlation peak of sound signals from two microphones. Furthermore, the control unit 12 obtains a cross-correlation peak of sound signals from another two microphones. The control unit 12 estimates the direction of arrival of the voice based on the multiple cross-correlation peaks calculated in this manner. That is, the control unit 12 selects two or more sets of multiple microphones and obtains multiple cross-correlation peaks. The estimated direction of arrival of the voice is expressed, for example, by a spatial vector.

[0017] The control unit 12 transmits the processed audio signal to the PC 40 via the communication I / F 11. The PC 40 is connected to another information processing device at a remote location via a network such as the Internet. The PC 40 transmits the audio signal received from the microphone 10 and the video signal received from the camera 30 to the information processing device on the far end side. The PC 40 may display the video signal received from the camera 30 on a display (not shown) of the PC 40 itself.

[0018] The PC 40 also receives a video signal and an audio signal from the information processing device on the far end. The PC 40 outputs the received video signal to a display device (not shown). The PC 40 also outputs the received audio signal to a speaker (not shown). This allows the image control system to function as a component of a remote conference system for holding a conference with a remote location.

[0019] The controller 20 is an example of an image control device of the present invention, and is a remote controller for operating the microphone 10 or the camera 30. The controller 20 includes a communication I / F 21, a control unit 22, a flash memory 23, a RAM 24, and a user I / F 25.

[0020] The control unit 22 controls the overall operation of the controller 20 by reading out the operation program 231 from the flash memory 23 into the RAM 24. It is not necessary to store the program in the flash memory 23 of the device itself. The control unit 22 may download the program from a server or the like and read it into the RAM 24 each time.

[0021] The control unit 22 accepts operations by the user via the user I / F 25. The user I / F 25 has at least a mute button. However, the user I / F 25 may also have other controls such as a volume change button or a power button.

[0022] 3 is a flowchart showing the operation of the image control system. The control unit 22 of the controller 20 first determines a mute operation (S11). The control unit 22 transmits mute information to the microphone 10 based on the determination result (S12). The mute operation is accepted via the user I / F 25. The mute operation includes a mute-on operation and a mute-off operation. If the control unit 22 determines a mute-on operation, it transmits the mute-on information to the microphone 10 as the mute information. If the control unit 22 determines a mute-off operation, it transmits the mute-off information to the microphone 10 as the mute information.

[0023] The microphone 10 receives mute information (S21). When the microphone 10 receives mute-on information, it stops outputting the sound signal acquired by the microphone unit 15 and changes to a mute state, and when the microphone 10 receives mute-off information, it resumes outputting the sound signal acquired by the microphone unit 15 and cancels the mute state.

[0024] Control unit 22 of controller 20 transmits control information for switching between the first state and the second state to camera 30 based on the determination result (S13). When control unit 22 determines that a mute-on operation has been performed, it transmits control information for switching to the first state to camera 30, and when it determines that a mute-off operation has been performed, it transmits control information for switching to the second state to camera 30.

[0025] The camera 30 receives control information (S31). The camera 30 switches the camera state based on the control information (S32). The camera state includes a first state in which the first image information is output and a second state in which the second image information is output.

[0026] Fig. 4 is a diagram showing an example of the first image information P1 output in the first state. The first state corresponds to a predetermined reference state. The reference state is, for example, a full-body shooting state in which a whole image of multiple users is shot. In the example of Fig. 4, the first image information P1 is framed so as to include images of both users u1 and u2.

[0027] Alternatively, the reference state may correspond to a state in which an image of a specific user is focused. The specific user is a moderator who proceeds with the conference. The specific user is set in advance by the PC 40. Alternatively, the reference state may be an initial state in which no framing processing such as panning, tilting, or zooming is performed.

[0028] The second state is a state in which framing processes such as panning, tilting, and zooming are performed to focus on the speaker. As an example, the camera 30 performs a process of recognizing the face of the speaker using a predetermined model using, for example, a neural network. The camera 30 pans, tilts, and zooms so that the image of the recognized speaker is at the center of the screen and the occupancy rate of the image of the speaker on the screen is a predetermined ratio (for example, 50%).

[0029] Fig. 5 is a diagram showing an example of the second image information P2 output in the second state. In the example of Fig. 5, the camera 30 recognizes the user u1 as the speaker. The second image information P2 is framed so that the image of the user u1 occupies 50% of the screen.

[0030] The camera 30 may receive directional information of the speaker's voice obtained by the microphone 10 and perform framing processing based on the directional information. The camera 30 may also perform processing to mask images of people other than the speaker or images of people not participating in the conference in the second state.

[0031] In this way, the image control system of this embodiment changes the camera state of the camera 30 in conjunction with the mute button of the controller 20. When the user performs a mute-on operation, the first image information output by the camera 30 is an image capturing the entire user. When the user performs a mute-off operation, the second image information output by the camera 30 is an image focusing on the speaker (user u1 in the example of FIG. 5).

[0032] The first image information and the second image information are transmitted to the information processing device on the far end side via the PC 40. The first image information or the second image information is displayed on a display of the information processing device on the far end side. Alternatively, the PC 40 may display the first image information and the second image information on a display (not shown) of its own device.

[0033] Typically, the mute off and mute on states are displayed on the GUI of the software of a remote conferencing system for holding conferences with remote locations. Therefore, as a comparative example, if the camera state does not change even when a user performs a mute operation, it is difficult for a user of the information processing device on the far end to notice that the state has changed from mute on to mute off, or from mute off to mute on.

[0034] However, in the image control system of this embodiment, when a user performs a mute-on operation, the image on the camera 30 becomes an image of the entire user. Users who see an image of the entire user can intuitively understand that no one wants to speak. In the image control system of this embodiment, when a user performs a mute-off operation, the image on the camera 30 becomes an image that focuses on the speaker (user u1 in the example of FIG. 5). Furthermore, users who see an enlarged image of user u1 can intuitively understand that user u1 wants to speak.

[0035] In this way, the user of the image control system of this embodiment can obtain a new customer experience in that he or she can clearly communicate to other conference participants whether or not he or she wishes to speak.

[0036] (Variation 1) Fig. 6 is a block diagram showing the configuration of an image control system according to Modification 1. Configurations common to Fig. 2 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 7 is a flowchart showing the operation of the image control system according to Modification 1. Configurations common to Fig. 3 are given the same reference numerals, and descriptions thereof will be omitted.

[0037] In the image control system according to the first modification, the camera 30 is directly connected to the PC 40. In the process of S13, the controller 20 transmits to the PC 40 control information for switching between the first state and the second state.

[0038] The PC 40 receives control information (S41). The PC 40 switches the state of the camera 30 based on the received control information (S42). As described above, the camera state includes a first state in which the first image information is output and a second state in which the second image information is output.

[0039] In this manner, the controller 20 may transmit control information to the camera 30 via an information processing device that receives the camera's video signal.

[0040] (Variation 2) Fig. 8 is a flowchart showing the operation of the image control system according to the modified example 2. The same components as those in Fig. 3 are denoted by the same reference numerals, and the description thereof will be omitted. The controller 20 according to the modified example 2 transmits control information to the camera 30 via the microphone 10.

[0041] Specifically, in the image control system according to the second modification, when the controller 20 transmits mute information to the microphone 10 in the processing of S12, the microphone 10 transmits control information for switching between the first state and the second state to the camera 30 (S23).

[0042] That is, in the second modification, the controller 20 does not transmit control information to the camera 30, but transmits mute information to the microphone 10, so that the microphone 10 transmits the control information to the camera 30.

[0043] When the camera 30 receives directional information of the speaker's voice obtained by the microphone 10 and performs framing processing based on the directional information, the microphone 10 may stop transmitting the directional information when transmitting control information for switching to the first state to the camera 30. This makes it possible to prevent conflict between the framing processing for focusing on the speaker based on the directional information and the framing processing for capturing an entire image of multiple users when the camera 30 switches to the first state.

[0044] (Variation 3) Fig. 9 is a block diagram showing the configuration of an image control system according to Modification 3. Configurations common to Fig. 2 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 10 is a flowchart showing the operation of the image control system according to Modification 3. Configurations common to Fig. 3 are given the same reference numerals, and descriptions thereof will be omitted.

[0045] The image control system according to the third modification includes a processor 50. The processor 50 is signal processing hardware (DSP: Digital Signal Processor). The processor 50 controls devices such as the microphone 10, a speaker (not shown), or a camera 30 connected to the image control system via a network, and performs signal processing such as routing, mixing, or effects on signals input to or output from each device.

[0046] The processor 50 may receive a sound signal acquired by the microphone unit 15 from the microphone 10 and perform beamforming directional processing. Alternatively, the processor 50 may analyze sound signals acquired from multiple microphones in the microphone unit 15 and estimate the direction of arrival of the sound. Alternatively, the processor 50 may receive a video signal captured by the camera 30 and perform framing processing.

[0047] The controller 20 is directly connected to the processor 50. The controller 20 transmits mute information to the processor 50 in the process of S12. The processor 50 receives the mute information (S51) and transmits the mute information to the microphone 10 (S52). Then, the processor 50 transmits control information for switching between the first state and the second state to the camera 30 (S53).

[0048] Thus, where the image control system includes a processor 50 , the controller 20 may transmit control information to the camera 30 via the processor 50 .

[0049] (Variation 4) Fig. 11 is a block diagram showing the configuration of an image control system according to Modification 4. Configurations common to Fig. 6 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 12 is a flowchart showing the operation of the image control system according to Modification 4. Configurations common to Fig. 7 are given the same reference numerals, and descriptions thereof will be omitted.

[0050] In the image control system according to the fourth modification, the camera 30 and the controller 20 are directly connected to the PC 40. The controller 20 transmits mute information to the PC 40 in the process of S12.

[0051] When the PC 40 receives the mute information (S61), it transmits the mute information to the microphone 10 (S62). Then, the PC 40 transmits control information for switching the camera 30 between the first state and the second state (S63).

[0052] In this manner, the controller 20 may transmit mute information to the microphone 10 and control information to the camera 30 via the PC 40 .

[0053] The description of the present embodiment should be considered as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention includes the scope equivalent to the claims.

[0054] For example, each device including the processor 50 does not need to be connected via a network, and may be connected via other communication lines such as USB. Alternatively, each device including the processor 50 may be connected wirelessly such as wireless LAN or Bluetooth (registered trademark). [Explanation of symbols]

[0055] 10: microphone, 11: communication I / F, 12: control unit, 13: flash memory, 14: RAM, 15: microphone unit, 20: controller, 21: communication I / F, 22: control unit, 23: flash memory, 24: RAM, 25: user I / F, 30: camera, 50: processor

Claims

1. Determines whether a mute operation has been performed, Transmitting control information to the camera for switching between the first state and the second state based on the result of the determination; causing a camera that outputs first image information in the first state and outputs second image information in the second state to output the first image information or the second image information based on the control information; Image control method.

2. the mute operation includes a mute-on operation and a mute-off operation, transmitting control information to the camera for switching to the first state when the mute-on operation is determined; transmitting control information to the camera for switching to the second state when the mute-off operation is determined; the first state corresponds to a predetermined reference state; The first image information includes an image corresponding to the reference state. The image control method according to claim 1 .

3. The reference state corresponds to a state in which an image of a plurality of users is taken in its entirety or an image of a specific user is focused on. The image control method according to claim 2 .

4. the second state corresponds to a state in which an image of a speaker is focused; The second image information includes an image focused on the speaker. The image control method according to claim 3.

5. transmitting the control information to the camera via a processor that receives a video signal from the camera; The image control method according to any one of claims 1 to 4.

6. transmitting the control information to the camera via an information processing device that receives a video signal of the camera; The image control method according to any one of claims 1 to 4.

7. Transmitting mute information to a microphone based on the result of the determination. The image control method according to any one of claims 1 to 4.

8. transmitting the control information to the camera via the microphone; The image control method according to claim 7.

9. The microphone obtains direction information of the speaker's voice and transmits the direction information to the camera; the camera focuses an image of the speaker based on the directional information; The image control method according to claim 7.

10. displaying the acquired first image information or the acquired second image information on a display device; The image control method according to any one of claims 1 to 4.

11. Determines whether a mute operation has been performed, Transmitting control information to the camera for switching between the first state and the second state based on the result of the determination; causing a camera that outputs first image information in the first state and outputs second image information in the second state to output the first image information or the second image information based on the control information; An image control device having a control unit.

12. the mute operation includes a mute-on operation and a mute-off operation, the control unit transmits control information to the camera for switching to the first state when the mute-on operation is determined, transmitting control information to the camera for switching to the second state when the mute-off operation is determined; the first state corresponds to a predetermined reference state; The first image information includes an image corresponding to the reference state. The image control device according to claim 11.

13. The reference state corresponds to a state in which an image of a plurality of users is taken in its entirety or an image of a specific user is focused on. The image control device according to claim 12.

14. the second state corresponds to a state in which an image of a speaker is focused; The second image information includes an image focused on the speaker. The image control device according to claim 13.

15. The control unit transmits the control information to the camera via a processor that receives a video signal from the camera. The image control device according to any one of claims 11 to 14.

16. The control unit transmits the control information to the camera via an information processing device that receives a video signal of the camera. The image control device according to any one of claims 11 to 14.

17. The control unit transmits mute information to a microphone based on the determination result. The image control device according to any one of claims 11 to 14.

18. The control unit transmits the control information to the camera via the microphone. The image control device according to claim 17.

19. The microphone obtains direction information of the speaker's voice and transmits the direction information to the camera; the camera focuses an image of the speaker based on the directional information; The image control device according to claim 17.

20. The control unit displays the acquired first image information or the acquired second image information on a display device. The image control device according to any one of claims 11 to 14.

21. Determines whether a mute operation has been performed, Transmitting control information to the camera for switching between the first state and the second state based on the result of the determination; causing a camera that outputs first image information in the first state and outputs second image information in the second state to output the first image information or the second image information based on the control information; A program that causes an information processing device to execute processing.

Images