Information processing device, information processing method, and program

Abstract

To appropriately perform control corresponding to a gesture of a subject in an imaging device regardless of an imaging state of the imaging device.SOLUTION: An information processing device includes detection means for detecting gesture information of a subject on the basis of a picked-up image of a second imaging device different from a first imaging device among a plurality of imaging devices including the first imaging device whose imaging state is controlled according to a gesture of the subject, acquisition means for acquiring control information for controlling the imaging state of the first imaging device associated with the gesture information detected by the detection means, and control means for controlling the imaging state of the first imaging device on the basis of the control information acquired by the acquisition means.SELECTED DRAWING: Figure 4

Description

Technical Field

[0004] ,

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, imaging systems that use a PTZ camera capable of pan-tilt-zoom control to image lecture scenes and the like and perform recording and distribution have been increasing. Also, due to labor saving, there is an increasing need for an automatic imaging system that uses a PTZ camera having an automatic imaging function to automatically image while detecting a lecturer as a subject to be tracked. Patent Document 1 discloses a technique for tracking a subject in a camera that performs tracking imaging of a subject, using both movement information indicating a change in the position of the subject detected by a main camera and movement information of the subject detected by a sub-camera that images the same subject. With this technique, it is possible to perform tracking imaging of the subject even in a situation where it is difficult to detect the movement of the subject with the main camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the imaging system as described above, it is possible to detect a gesture caused by the movement of the hand or arm of the subject by estimating the posture of the subject from the captured image. If the imaging range of the camera and the like are automatically controlled according to the detected gesture, the lecturer as the subject can control the camera as intended by the gesture, and it is possible to perform desired imaging without a cameraman. However, if a gesture-controlled camera is capturing a close-up of the subject performing the gesture, or capturing an image of something other than the subject performing the gesture, the arm or hand performing the gesture will not be included in the captured image. Therefore, it is not possible to detect the gesture from the captured image, and camera control based on gesture information cannot be performed.

[0005] Therefore, the present invention aims to appropriately control the imaging device in response to the gestures of the subject, regardless of the imaging state of the imaging device. [Means for solving the problem]

[0006] To solve the above problems, one aspect of the information processing apparatus according to the present invention includes: detection means for detecting gesture information of a subject based on an image captured by a second imaging device different from the first imaging device, among a plurality of imaging devices including a first imaging device whose imaging state is controlled in accordance with the gesture of the subject; acquisition means for acquiring control information for controlling the imaging state of the first imaging device, which is associated with the gesture information detected by the detection means; and control means for controlling the imaging state of the first imaging device based on the control information acquired by the acquisition means. [Effects of the Invention]

[0007] According to the present invention, control in response to the gestures of the subject in the imaging device can be appropriately performed regardless of the imaging state of the imaging device. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example configuration of the imaging system in this embodiment. [Figure 2] A diagram showing an example of camera installation. [Figure 3] A diagram showing an example of the hardware configuration of a gesture detection device. [Figure 4] A block diagram showing an example of the functional configuration of a gesture detection device. [Figure 5]A flowchart illustrating an example of the processing performed by a gesture detection device. [Figure 6] A diagram showing examples of gesture-controlled cameras and gesture-detection cameras. [Figure 7] A block diagram showing an example of the functional configuration of a gesture detection device. [Figure 8] A flowchart illustrating an example of the processing performed by a gesture detection device. [Figure 9] A diagram illustrating an example of changing the camera for gesture detection. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the attached drawings. The embodiments described below are merely examples of means for realizing the present invention, and should be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. The present invention is not limited to the embodiments described below.

[0010] (First embodiment) First, a first embodiment of the present invention will be described. [System Configuration] Figure 1 is a diagram showing an example of the overall configuration of the imaging system 1000 according to this embodiment. As shown in Figure 1, the imaging system 1000 comprises a plurality of network cameras (hereinafter simply referred to as "cameras") 100, a network 200, a gesture detection device 300, and a multi-camera control device 400. The plurality of cameras 100, the gesture detection device 300, and the multi-camera control device 400 are each connected via the network 200.

[0011] As shown in FIG. 2, a plurality of cameras 100 are installed facing an imaging region 500 and image a predetermined region within the imaging region 500. In the present embodiment, an example will be described in which the imaging region 500 is a classroom where lectures are given, and the cameras 100 automatically image and record a lecture scene by a subject 601 who is a lecturer. However, the imaging region 500 is not limited to a classroom and may be an imaging studio, a stage where performances are held, or the like.

[0012] The plurality of cameras 100 are imaging devices that can be remotely controlled from a gesture detection device 300 and a multi-camera control device 400 through a network 200. Further, the camera 100 may have an automatic imaging function of automatically imaging a subject 601 based on an instruction of an automatic imaging method (automatic imaging instruction) from the gesture detection device 300 and the multi-camera control device 400. For example, the camera 100 can automatically track and image the subject 601 by performing PTZ (pan, tilt, zoom) control according to the movement of the subject 601 based on the above automatic imaging instruction. Here, the above automatic imaging instruction includes an instruction regarding the subject to be the imaging target for tracking imaging, an instruction regarding the imaging composition of the subject to be the imaging target, and the like. The camera 100 identifies the subject to be the imaging target by, for example, human body detection or face authentication based on the automatic imaging instruction, and performs PTZ control so that the subject appears at a specific position (for example, the center of the image) in the image with a specific size (for example, a full body shot).

[0013] Also, the plurality of cameras 100 can transmit imaging data and imaging information in response to requests from the gesture detection device 300 and the multi-camera control device 400. Here, the imaging information is information regarding the imaging state of the camera 100 and includes information regarding the subject during imaging, imaging composition, PTZ values, and the like.

[0014] In the present embodiment, the case where the camera 100 has an automatic imaging function will be described. However, the camera 100 may be configured such that an automatic imaging device that detects a subject from an imaging image and issues a PTZ control instruction is externally attached.

[0015] Also, when the imaging area 500 is a classroom where lectures are held, in the imaging system 1000, the camera 100 with an automatic imaging function may be other than the camera that tracks and images the lecturer. For example, the imaging system 1000 may include a camera 100 that images students attending lectures as subjects, or a camera 100 that images a whiteboard, blackboard, monitor, screen, etc. other than people. Also, in the case of an imaging system 1000 using a plurality of PTZ controllable cameras 100, automatic imaging may be performed while changing the imaging target of each camera 100 according to the situation of the imaging area 500. Furthermore, in addition to the camera that realizes the automatic imaging function by PTZ control, the imaging system 1000 may include one or more fixed cameras as overview cameras that always image the entire imaging area 500 or most of it. In this case, the multi-camera control device 400 can grasp the situation of the imaging area 500 based on the imaging data of the overview camera and give an automatic imaging instruction to the camera 100 that performs automatic imaging.

[0016] The network 200 can be realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark). Note that the network 200 may be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), or a combination thereof. The network 200 may have any communication standard, scale, and configuration as long as it can communicate between the camera 100, the gesture detection device 300, and the multi-camera control device 400.

[0017] The gesture detection device 300 is composed of, for example, a personal computer (PC) or a server device, and has a gesture detection function and a camera control function. Here, the gesture detection function is a function that detects the gesture of a subject from the image captured by a specific camera 100. The camera control function is a function that instructs the same or a different specific camera 100 as the camera 100 that detected the gesture to perform control (gesture control) according to the detected gesture.

[0018] One example of a method for detecting a subject's gestures from captured images is to detect the human body from the captured image, recognize body parts such as the head, neck, shoulders, elbows, and hands, and then estimate the subject's pose. By identifying the subject's pose, it is possible to detect gestures such as raising the hands or extending both arms to the sides. However, the method of detecting gestures from captured images is not limited to the above; for example, the detection method may be changed depending on the gesture to be detected, such as recognizing the joints of the fingers to detect more detailed finger gestures.

[0019] Furthermore, the gesture detection device 300 can receive user input, select a camera to perform gesture control, and register the gestures to be detected and the corresponding camera controls. Examples of gestures to be detected include the subject's pose (e.g., raising a hand) or changes in pose (e.g., raising a hand and then lowering it). Furthermore, the system may detect a gesture if the same pose is maintained for a certain period of time or longer; in this case, the detection duration frame count may be finely configurable. Additionally, the gestures to be detected may be combinations of multiple pre-registered gestures.

[0020] Furthermore, camera control methods that support gestures include controlling the automatic imaging method, preset control that performs PTZ control so that the imaging range becomes a pre-registered imaging range, and controlling the imaging range by changing the PTZ value during gesture detection. Furthermore, the gesture detection device 300 may be configured to register multiple patterns of correspondence between gestures and camera control, depending on the camera being controlled by the gesture.

[0021] The multi-camera control device 400 can be configured, for example, with a PC, server, or tablet terminal. The multi-camera control unit 400 can issue imaging instructions to multiple cameras 100 and manage the imaging status of the multiple cameras 100. Here, the imaging instructions for camera 100 include automatic imaging instructions for tracking and capturing images of a specific person, imaging instructions for capturing overhead images, or imaging instructions for capturing a specific subject with a specific composition.

[0022] Furthermore, the multi-camera control device 400 may also be equipped with image analysis and imaging decision functions. In this case, the multi-camera control device 400 can, for example, analyze images captured by an overhead camera among the multiple cameras 100 to detect the positions and movements of instructors and students in the classroom, which is the imaging area 500, if it is a lecture recording. Based on the detection results, the multi-camera control device 400 can then decide how to image the imaging area 500 and issue an automatic imaging instruction to the camera among the multiple cameras 100 that has an automatic imaging function.

[0023] The configuration of the imaging system 1000 is not limited to the configuration shown in Figure 1. For example, the number of cameras 100 connected to the network 200 is not limited to the number shown in Figure 1, but can be two or more. Also, multiple gesture detection devices 300 may be connected to the network 200. Furthermore, the gesture detection devices 300 and the multi-camera control device 400 are not limited to physically independent devices, but may be configured as the same device (e.g., a PC). Alternatively, instead of providing a separate gesture detection device 300 from the camera 100, the camera 100 may be equipped with a gesture detection function, and the gesture detection results may be communicated between multiple cameras 100 to enable camera control that corresponds to gesture detection.

[0024] [Hardware configuration] Next, the hardware configuration of each device in this embodiment will be described. Figure 3 shows an example of the hardware configuration of the gesture detection device 300 in this embodiment. The gesture detection device 300 includes a CPU 301, ROM 302, RAM 303, HDD (hard disk drive) 304, input unit 305, display unit 306, and communication unit 307. The gesture detection device 300 may further include configurations other than those described above. The CPU 301 comprehensively controls the operation of the gesture detection device 300. The ROM 302 has a storage area for programs and data necessary for the CPU 301 to execute processing. These programs may also be stored in the HDD 304 or a removable storage medium (not shown). The RAM 303 functions as the main memory, work area, etc., of the CPU 301. When executing processing, the CPU 301 loads the necessary programs, etc., from the ROM 302 into the RAM 303 and executes these programs, etc., to realize various functional operations.

[0025] HDD304 can be used as a persistent storage area for the OS, various programs, and various data, as well as for short-term storage of various data. Note that other auxiliary storage devices such as SSDs may be used instead of HDD304. The input unit 305 includes, for example, an operation unit such as a keyboard, mouse, joystick, or touch panel, and receives various instructions from the user via the operation unit and inputs them to the CPU 301. The display unit 306 is equipped with a monitor such as a liquid crystal display (LCD) and displays a GUI for the user to operate the gesture detection device 300. The CPU 301 can operate as a display control unit that controls the display unit 306. The communication unit 307 transmits and receives data to and from external devices such as the camera 100 and the multi-camera control device 400 via the network 200.

[0026] In this embodiment, the case in which the gesture detection device 300 comprises an input unit 305 and a display unit 306 is described, but at least one of the input unit 305 and the display unit 306 may exist as a separate device outside the gesture detection device 300. Some or all of the functions of the gesture detection device 300 are realized by the CPU 301 executing a program stored in the ROM 302 or HDD 304. However, at least some of the functions of the gesture detection device 300 may operate as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 301.

[0027] Furthermore, the multi-camera control device 400 can have the same hardware configuration as the automatic imaging instruction device 300. Furthermore, the hardware configuration of camera 100 can also conform to the hardware configuration shown in Figure 3. However, in the case of camera 100, an imaging unit is provided instead of a display unit 306. Here, the imaging unit captures images of the subject and generates captured images. The imaging unit can be configured to include, for example, an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device), an A / D converter, a development processing unit, etc. Also, in the case of camera 100, the input unit 305 consists of a power button, setting buttons, etc., and the operator of camera 100 can give instructions to camera 100 via the input unit 305. Furthermore, some or all of the functions of camera 100 are realized by the execution of a program by the CPU of camera 100, which corresponds to CPU 301. However, at least some of the functions of camera 100 may operate as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU.

[0028] In this embodiment, we will describe a case in which the gesture detection device 300 detects the gestures of a subject from the image captured by the camera 100 and functions as an information processing device that performs camera control according to the detected gestures. However, a multi-camera control device 400 or a general PC that is communicatively connected to the camera 100 may also operate as the information processing device, or one of the multiple cameras 100 may operate as the information processing device.

[0029] [Functional Configuration] Figure 4 is a block diagram showing an example of the functional configuration of the gesture detection device 300. The gesture detection device 300 comprises a gesture management unit 311, an image acquisition unit 312, a subject detection unit 313, a gesture detection unit 314, and a camera control unit 315. Each element shown in Figure 4 can be realized by the CPU 301 of the gesture detection device 300 executing the corresponding processing program. However, at least some of the elements shown in Figure 4 may be configured to be realized by dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 301. In this embodiment, we will describe a case where the gesture detection device 300 has the functions shown in Figure 4, but some of the functions may be provided by other devices.

[0030] The gesture management unit 311 manages gesture detection camera information and gesture control camera information from among the multiple cameras 100 in the imaging system 1000. Here, the gesture detection camera is a camera that captures images used for gesture detection, and the gesture control camera is a camera whose imaging state is controlled according to the gesture information detected from the images captured by the gesture detection camera. The gesture detection camera and the gesture control camera may be the same camera or different cameras. In this embodiment, the gesture detection camera and the gesture control camera are selected by the user, but the gesture detection device 300 may determine the gesture detection camera. In this case, the gesture detection device 300 may, for example, acquire the imaging status of multiple cameras 100 from the multi-camera control device 400 and determine the gesture detection camera based on the acquired imaging status.

[0031] Furthermore, the gesture management unit 311 manages the gesture information to be detected and the correspondence information, which associates the gesture information with the camera control information, as management information. The above management information may be stored in advance, or it may be made available for user registration and modification. Furthermore, the gesture management unit 311 may manage the above management information in multiple patterns depending on the imaging state of the gesture control camera. Note that the above management information may also be managed by an external device, in which case the gesture management unit 311 may acquire the management information from the external device.

[0032] The image acquisition unit 312 receives imaging data from the gesture detection camera among the multiple cameras 100, according to the gesture detection camera information managed by the gesture management unit 311. The image processing unit 312 then performs image processing on the received imaging data and transmits the processed image to the subject detection unit 313. The subject detection unit 313 analyzes the image received from the image acquisition unit 312, detects a person who is the subject of the gesture, estimates the posture of the detected person, and transmits the estimated posture information to the gesture detection unit 314. If only the gesture of a specific person is to be enabled, the subject detection unit 313 identifies the person based on pre-registered person characteristics and transmits the posture information of only the identified person to the gesture detection unit 314. On the other hand, if gestures of all people in the image are to be enabled, the subject detection unit 313 transmits person information such as the registered person's ID and person characteristics, along with the posture information, to the gesture detection unit 314.

[0033] The gesture detection unit 314 performs gesture detection using the posture information of a person received from the subject detection unit 313, based on the gesture information to be detected obtained from the gesture management unit 311. The gestures to be detected are not particularly limited; for example, it may be the pose of a subject in one frame, the detection of the same pose continuing for multiple frames, or the detection of a change in pose between multiple frames. When performing gesture detection using posture information from multiple frames, the gesture detection shall be performed using the posture information of the person based on the person information received from the subject detection unit 313. When the gesture detection unit 314 detects a gesture to be detected, it notifies the camera control unit 315 of the detected gesture information.

[0034] When the camera control unit 315 receives notification of gesture information from the gesture detection unit 314, it refers to the corresponding information obtained from the gesture management unit 311 and obtains camera control information corresponding to the gesture information detected by the gesture detection unit 314. The camera control unit 315 may also obtain the camera control information corresponding to the gesture information detected by the gesture detection unit 314 from an external device. Then, the camera control unit 315 transmits a camera control instruction to the gesture control camera based on the camera control information, according to the gesture control camera information managed by the gesture management unit 311.

[0035] [Processing flow] Next, using Figure 5, the processing flow for realizing the gesture detection function and camera control function in the gesture detection device 300 of this embodiment will be described. This section describes the processing flow when an overhead camera is used as a gesture detection camera, and the gestures of the subject 601 (the lecturer) are detected from the images captured by the overhead camera, and the gesture control camera is controlled according to the detected gestures while recording the lecture.

[0036] Figure 6 is an explanatory diagram of the gesture detection camera and the gesture control camera. Camera 100A is a focus camera that tracks and images the subject 601, and is assumed to be imaging the imaging range 510 including the subject 601. Camera 100B is an overhead camera that images the entire range in which the subject 601 may move, and is assumed to be imaging the imaging range 520. The gesture detection device 300 can perform each of the processes shown in Figure 5 by having the CPU 301 in Figure 3 read and execute the necessary program. Hereafter, the letter S will represent a step in the flowchart.

[0037] In S1 of Figure 5, the gesture management unit 311 selects a gesture control camera from among the multiple cameras 100. Specifically, the gesture management unit 311 acquires information on the gesture control camera specified by the user operation. Alternatively, the gesture management unit 311 may acquire the imaging status of the multiple cameras 100 from the multi-camera control device 400 and select a gesture control camera based on the acquired imaging information. Alternatively, the gesture management unit 311 may acquire information on the gesture control camera selected by the multi-camera control device 400 based on the imaging status of the multiple cameras 100. As the gesture control camera, for example, a camera of interest that is tracking and imaging the subject 601 may be selected. In the example shown in Figure 6, it is assumed that camera 100A is selected as the gesture control camera.

[0038] In S2, the gesture management unit 311 selects a gesture detection camera from among the multiple cameras 100. Specifically, the gesture management unit 311 acquires information on the gesture detection camera specified by the user. Alternatively, the gesture management unit 311 may acquire the imaging status of the multiple cameras 100 from the multi-camera control device 400 and select a gesture detection camera based on the acquired imaging information. Alternatively, the gesture management unit 311 may acquire information on the gesture detection camera selected by the multi-camera control device 400 based on the imaging status of the multiple cameras 100. As the gesture detection camera, for example, an overhead camera that captures the entire range in which the subject 601 may move may be selected. In the example shown in Figure 6, it is assumed that camera 100B is selected as the gesture detection camera.

[0039] In S3, the gesture detection device 300 determines whether or not the target gesture has been detected based on the image captured by the gesture detection camera. In S3, the image processing unit 312 first acquires the image data from the camera 100B, which was selected as the gesture detection camera in S2, processes the image, and transmits it to the subject detection unit 313. The subject detection unit 313 detects the posture information of the subject 601 based on the image acquired from the image processing unit 312. Then, the gesture detection unit 314 uses the posture information detected by the subject detection unit 313 to perform gesture detection based on the target gesture information managed by the gesture management unit 311. If the gesture detection unit 314 determines that the target gesture has been detected, the gesture detection device 300 proceeds to S4; otherwise, it proceeds to S5.

[0040] In S4, the camera control unit 315 refers to the correspondence information between gesture information and camera control information managed by the gesture management unit 311 and determines the camera control information corresponding to the gesture information detected in S3. Then, based on the determined camera control information, the camera control unit 315 issues a camera control instruction to the camera 100A selected as the gesture control camera in S1. In S5, the gesture detection device 300 determines whether or not it has received an instruction from the user to end gesture control. If there is no instruction to end gesture control, it repeats the process from S3. If there is an instruction to end gesture control, it terminates the process shown in Figure 5.

[0041] Through the above processing flow, the gestures of the subject can be detected from the image captured by the overhead camera, which is a gesture detection camera separate from the gesture control camera. Based on the detected gesture information, the imaging state of the focus camera, which is the gesture control camera, can be controlled. Therefore, appropriate gesture control can be performed regardless of the imaging state of the gesture control camera.

[0042] As described above, the gesture detection device 300 in this embodiment controls the imaging state of the gesture control camera according to the gesture of the subject. At this time, the gesture detection device 300 detects the gesture information of the subject based on the image captured by a gesture detection camera different from the gesture control camera, and acquires camera control information associated with the detected gesture information. Then, the gesture detection device 300 controls the imaging state of the gesture control camera based on the acquired camera control information. Thus, since gesture detection is performed using images captured by a camera different from the gesture control camera, gesture information can be appropriately acquired even if the gesture control camera does not capture the body parts of the subject necessary for gesture detection. Therefore, the imaging state of the gesture control camera can be appropriately controlled according to the subject's gestures.

[0043] Here, as the gesture detection camera, one of the multiple cameras 100 can be selected to capture an overhead image of the area where the subject may move. By using an overhead camera as the gesture detection camera in this way, the subject's gestures can be detected appropriately.

[0044] Furthermore, the gesture detection device 300 manages correspondence information that associates the subject's gesture information with camera control information, and can obtain camera control information associated with the gesture information detected by gesture detection by referring to this correspondence information. Therefore, it is possible to appropriately control the camera in accordance with the subject's gestures. In addition, by configuring the above correspondence information to be registerable and modifiable by the user, it becomes possible to control gestures in accordance with the intentions of the subject performing the gesture.

[0045] As described above, in this embodiment, camera control can be appropriately performed in response to the subject's gestures, regardless of the imaging state of the gesture-controlled camera.

[0046] (Second Embodiment) Next, a second embodiment of the present invention will be described. The first embodiment described above described a case where the gesture detection camera and the gesture control camera are always different cameras. The second embodiment describes a case where the gesture detection camera and the gesture control camera are usually the same camera. Then, an example is described in which the gesture detection camera is changed to a different camera when gesture detection of the subject becomes impossible due to a change in the field of view of the gesture control camera, etc., while gesture detection is being performed based on the image captured by the gesture control camera.

[0047] The imaging system in the second embodiment can have the same configuration as the imaging system 1000 shown in Figure 1. Furthermore, the hardware configuration of the gesture detection device 300 can be the same as the hardware configuration shown in Figure 3.

[0048] [Functional Configuration] Figure 7 is a block diagram showing an example of the functional configuration of the gesture detection device 300 in the second embodiment. In Figure 7, parts having the same configuration as those shown in Figure 4 are denoted by the same reference numerals as in Figure 4, and the following description will focus on the parts with different configurations. The gesture detection feasibility determination unit 321 receives the posture information of the subject in the image captured by the gesture detection camera via the image acquisition unit 312 and the subject detection unit 313, and then determines whether or not gesture detection is possible from the image captured by the gesture detection camera. If the gesture detection feasibility determination unit 321 determines that gesture detection is not possible, it notifies the gesture detection camera change unit 322 of this fact.

[0049] Whether or not gesture detection is possible can be determined based on the gesture information to be detected, which is managed by the gesture management unit 311. When the target of detection is a gesture involving arm or hand movement, whether or not gesture detection is possible is determined by whether or not the subject's arm or hand can be detected from the captured image. If the target of detection includes foot movement, whether or not gesture detection is possible is determined by whether or not the subject's foot can be detected from the captured image. In this way, the determination method is changed according to the gesture information to be detected.

[0050] Furthermore, if the gesture detection camera is performing automatic image capture, the system may determine whether gesture detection is possible based on the automatic image capture status. For example, if tracking image capture is performed using an automatic zoom adjustment function that maintains a constant size of the subject's face or body in the captured image, the system may determine whether gesture detection is possible based on the zoom mode. Zoom modes include, for example, a full-body mode that captures the entire body, an upper-body mode that captures only the upper body, and a close-up face mode that captures a close-up of the face. When detecting gestures caused by arm or hand movements, gesture detection is determined to be possible in full-body mode or upper-body mode, and not possible in close-up face mode. However, the imaging state used for determining whether or not a gesture can be detected only needs to be such that it can be identified whether the captured image contains the body parts necessary for gesture detection, and is not limited to the zoom mode of the automatic zoom adjustment function described above. The imaging state of the gesture detection camera can be obtained from the gesture detection camera or the multi-camera control device 400.

[0051] Furthermore, the gesture information to be detected may change depending on the imaging state of the gesture control camera. For example, during imaging in a specific mode, only specific camera controls may be allowed or not allowed, or camera control by gestures may not be accepted for a certain period after a specific mode change, so that only gestures corresponding to executable camera controls may be detected. In other words, the gesture management unit 311 may manage correspondence information that associates the gesture information to be detected, which is determined according to the imaging state of the gesture control camera, with camera control information. In that case, the criteria for determining whether or not gesture detection is possible in the gesture detection feasibility determination unit 321 shall change depending on the imaging state of the gesture control camera. Furthermore, if a camera other than the gesture control camera is used as the gesture detection camera, the gesture detection feasibility determination unit 321 will always determine whether gesture detection is possible not only from the gesture detection camera but also from the image captured by the gesture control camera. In this case, the gesture detection feasibility determination unit 321 will determine whether gesture detection is possible from the image captured by the gesture control camera via the image acquisition unit 312 and the subject detection unit 313.

[0052] When the gesture detection camera change unit 322 receives notification from the gesture detection feasibility determination unit 321 that gesture detection is not possible, it obtains the imaging status of the multiple cameras 100 included in the imaging system 1000 from the multi-camera control device 400. The gesture detection camera change unit 322 then determines a new gesture detection camera from among the multiple cameras 100 and notifies the gesture management unit 311. At this time, the gesture management unit 311 receives notification from the gesture detection camera change unit 322 and saves (updates) the gesture detection camera information.

[0053] Furthermore, once the gesture detection camera changing unit 322 determines a new gesture detection camera, it issues a camera control instruction to the new gesture detection camera via the camera control unit 315, instructing it to change the imaging range to an imaging range capable of detecting gestures. The camera control instruction method involves specifying information about the subject to be included in the imaging range, such as subject information and information about the part of the body targeted for gesture detection, and then issuing a PTZ control instruction so that the imaging range becomes an imaging range where gesture detection is possible. However, if an overhead camera, such as camera 100B shown in Figure 6 of the first embodiment described above, is used as the new gesture detection camera, it may be considered unnecessary to change the imaging range, and therefore no camera control instructions may be issued. Alternatively, camera control instructions may be issued to image the subject only when the subject is outside the imaging range.

[0054] [Processing flow] Next, using Figure 8, the processing flow for realizing the gesture detection function and camera control function in the gesture detection device 300 in this embodiment will be described. Here, the processing flow for changing the gesture detection camera when it is determined that gesture detection is not possible from the image captured by the gesture control camera will be described. In Figure 8, the parts that perform the same processing as the gesture detection device 300 in Figure 5 are assigned the same step numbers as in Figure 5, and the following explanation will focus on the parts that differ in processing. Furthermore, in the following explanation, if the gesture control camera and the gesture detection camera are the same camera, the camera that performs gesture detection will also be referred to as the gesture control camera.

[0055] Figure 9 is a diagram illustrating an example of changing the gesture detection camera in this embodiment. Figure 9 shows an image of changing the gesture detection camera and the transition of the imaging range of the gesture detection camera in an imaging system that records lectures while controlling the camera in accordance with the gestures of the subject 601, who is the lecturer. Camera 100A is a focus camera that tracks and images the subject 601, and at the time shown in Figure 9(A), it is capturing an imaging range 511 that includes the entire body of the subject 601. Camera 100B is capturing an imaging range 521 that includes the monitor 602. In this imaging system shown in Figure 9, camera 100B is not a panoramic camera that always captures the whole scene, but is intended to be used as a multi-purpose camera that captures different targets depending on the situation.

[0056] In S11 of Figure 8, the gesture management unit 311 selects a gesture control camera from among the multiple cameras 100. Specifically, the gesture management unit 311 acquires information on the gesture control camera specified by the user. In this embodiment, gesture detection is normally performed based on the image captured by the gesture control camera, so the gesture management unit 311 also acquires and stores the information on the gesture control camera specified by the user as gesture detection camera information. As the gesture control camera, for example, the camera of interest that is tracking and imaging the subject 601 may be selected. In the example shown in Figure 9(A), camera 100A is selected as the gesture control camera, and similarly, the gesture detection camera is also set to camera 100A.

[0057] Next, in S12, the gesture detection feasibility determination unit 321 determines whether gesture detection is possible based on the pose information of the subject detected from the image captured by the gesture control camera selected in S11. If the gesture detection feasibility determination unit 321 determines that gesture detection is not possible, it proceeds to S13; if it determines that gesture detection is possible, it proceeds to S3.

[0058] At the point shown in Figure 9(A), the camera 100A is capturing images of the entire body of the subject 601 using the automatic zoom adjustment function. If the target of detection is gestures caused by arm or hand movements, it is possible to detect the subject 601's gestures based on the images captured by camera 100A. However, as shown in Figure 9(B), when the zoom mode of the automatic zoom adjustment function in camera 100A is changed to face-up mode, the camera 100A's imaging range 512 does not include the subject 601's arms or hands. Therefore, it is not possible to detect the subject 601's gestures based on the images captured by camera 100A. Therefore, in this case, the gesture detection feasibility determination unit 321 determines that gesture detection is not possible with the camera 100A, which is the gesture control camera, and proceeds to S13.

[0059] In S13, the gesture detection camera change unit 322 determines a new gesture detection camera and notifies the gesture management unit 311 of the change in the gesture detection camera. The gesture management unit 311 registers the new gesture detection camera notified by the gesture detection camera change unit 322. As a result, the gesture detection camera is changed. Here, the gesture detection camera is assumed to have been changed from camera 100A to the other camera 100B.

[0060] If the gesture detection camera is changed in S13, the gesture detection camera information may be presented to the subject 601 so that the subject 601 performing the gesture can see the gesture detection camera. Methods of presentation include presentation using camera tally control, or, if the captured image is displayed to the subject 601, presentation via screen display. Furthermore, the gesture detection camera information may be displayed to the subject 601 at all times while gesture detection is being accepted. Additionally, the gesture detection camera information may be displayed to the subject 601 only when the gesture detection camera is different from the gesture control camera. By presenting gesture detection camera information to the subject 601, the subject 601 can recognize which camera's image is currently being used for gesture detection and determine which camera it is preferable to perform the gesture towards.

[0061] Next, in S14, the gesture detection camera modification unit 322 issues a camera control instruction to the new gesture detection camera modified in S13, instructing it to change the imaging range to an imaging range capable of detecting gestures. Figures 9(C-1) and 9(C-2) show examples in which the gesture detection camera is changed from camera 100A to camera 100B, and the imaging range of camera 100B is changed to an imaging range that enables gesture detection of subject 601. Figure 9(C-1) shows an example in which the imaging range of camera 100B is changed to an imaging range 522a that can capture the entire body of subject 601. Figure 9(C-2) shows an example in which the imaging range of camera 100B is changed to an imaging range 522b that can capture both the monitor 602, which camera 100B originally captured, and the entire body of subject 601.

[0062] Thus, the image capture range after the modification of the new gesture detection camera may be determined specifically for gesture detection, or it may be an expansion of the previous image capture range. Furthermore, the method for determining the image capture range after the modification of the new gesture detection camera is not limited to the above. Furthermore, as shown in Figure 9(C-2), when expanding the imaging range of camera 100B for gesture detection, a portion of the image corresponding to the original imaging range 521 may be extracted from the image captured by camera 100B and used as the image captured by camera 100B. Furthermore, if the imaging range of the gesture detection camera is changed in S14, the system may display information on the screen showing the captured image or record it as information in the recorded video so that the user can see that the imaging range for gesture detection has been changed. By displaying information to the user or recording it in the captured image so that the user can see that the video as it is is likely to be unsuitable for distribution.

[0063] Furthermore, if the gesture detection device 300 determines in S5 that there is no instruction from the user to end gesture control, it proceeds to S15. Then, in S15, the gesture management unit 311 determines whether the gesture control camera and the gesture detection camera are the same camera. If it determines that they are the same camera, it returns to S12; otherwise, it proceeds to S16.

[0064] In S16, the gesture detection feasibility determination unit 321 determines whether or not gesture detection is possible from the image captured by the gesture control camera. If the gesture detection feasibility determination unit 321 determines that gesture detection is possible, it proceeds to S17; if it determines that gesture detection is not possible, it proceeds to S19. For example, if the zoom mode of camera 100A is changed again from a close-up face mode as shown in Figures 9(C-1) and 9(C-2) to a full-body mode as shown in Figure 9(A) due to gesture control of the subject 601, it is determined that gesture detection is possible.

[0065] In S17, if the gesture detection camera modification unit 322 changed the imaging range of the gesture detection camera in S14 to perform gesture detection, it returns the imaging range of the gesture detection camera to the imaging range before the change. The gesture detection camera modification unit 322 may also issue an instruction to the gesture detection camera to terminate control of the imaging range for gesture detection. Next, in S18, the gesture detection camera change unit 322 registers the gesture detection camera information with the gesture management unit 311 so that the gesture detection camera is changed back to the gesture control camera, and then returns to S12.

[0066] In S19, the gesture detection feasibility determination unit 321 determines whether gesture detection is possible from the image captured by the gesture detection camera. If the gesture detection feasibility determination unit 321 determines that gesture detection is possible, it returns to S3. If it determines that gesture detection is not possible, it determines that the gesture detection camera needs to be changed and returns to S13. If it is determined in S19 that gesture detection is impossible, the system may decide that it is necessary to change the imaging range of the gesture detection camera to an imaging range that enables gesture detection and return to S14. Furthermore, in S12, S16, or S19, when determining whether or not gesture detection is possible from the captured image, the determination may be made whether or not gesture detection is possible if the state of being able or impossible to detect gestures continues for a certain period of time.

[0067] With the above processing flow, gesture control is normally performed based on the image captured by the gesture control camera, and only when gesture detection is impossible from the image captured by the gesture control camera can the gesture detection camera be changed to perform gesture control.

[0068] As described above, the gesture detection device 300 in this embodiment determines whether or not gesture information of the subject can be detected from the image captured by the gesture control camera. If the gesture detection device 300 determines that gesture information cannot be detected from the image captured by the gesture control camera, it detects the gesture information of the subject based on the image captured by a gesture detection camera different from the gesture control camera. On the other hand, if the gesture detection device 300 determines that gesture information can be detected from the image captured by the gesture control camera, it detects the gesture information of the subject based on the image captured by the gesture control camera.

[0069] Thus, only when gesture detection is not possible from the image captured by the gesture control camera, gesture detection may be performed using the image captured by a gesture detection camera different from the gesture control camera. This allows the gesture control camera and the gesture detection camera to be the same camera under normal circumstances, and gestures made by the subject toward the gesture control camera can be appropriately detected from the image captured by the gesture control camera.

[0070] Here, the determination of whether or not gesture detection is possible from the image captured by the gesture control camera can be made based on at least one of the image captured by the gesture control camera and the imaging state. In this case, it is possible to identify whether or not the image captured by the gesture control camera contains the body parts necessary for gesture detection, and to appropriately determine whether or not gesture detection is possible from the image captured by the gesture control camera.

[0071] In the example shown in Figure 9, the imaging system has two cameras, and if gesture detection becomes impossible from the image captured by camera 100A, which is the gesture control camera, the other camera 100B is selected as the gesture detection camera. However, if the imaging system has two or more cameras other than the gesture control camera, one of them shall be selected and designated as the gesture detection camera. When selecting a gesture detection camera from multiple cameras, a new gesture detection camera can be selected based on the imaging status of each camera. For example, an overhead camera, a camera imaging the subject performing the gesture, a camera imaging the area closest to the subject performing the gesture, or a camera imaging the area with the lowest priority (such as the subject) can be selected as a gesture detection camera. In other words, the goal is to select a camera as a gesture detection camera that does not require changes to its imaging range, a camera that undergoes minimal changes to its imaging range, and a camera that has little impact on imaging.

[0072] Furthermore, when selecting a gesture detection camera from multiple cameras, it is also possible to select a new gesture detection camera based on the streaming status of each camera. For example, if image capture is performed simultaneously with image streaming, it is undesirable to change the imaging range, so a camera that is not currently streaming may be selected as the gesture detection camera. Furthermore, when selecting a gesture detection camera from multiple cameras, it is also possible to select a new gesture detection camera based on the arrangement of each camera. For example, assuming that the subject will perform a gesture towards the gesture control camera, the camera located closest to the gesture control camera may be selected as the gesture detection camera. Note that these are not the only methods for selecting a gesture detection camera.

[0073] Furthermore, as in the first embodiment described above, the method for selecting the gesture detection camera described above is also applicable when gesture detection is always performed from images captured by a gesture detection camera different from the gesture control camera. In other words, the gesture detection device 300 may determine the gesture detection camera based on at least one of the imaging state, distribution state, and placement state of the multiple cameras 100. In this case, the gesture detection device 300 can change the imaging range of the determined gesture detection camera to an imaging range in which gesture information of the subject can be detected from the image captured by the gesture detection camera, similar to the second embodiment. At this time, the gesture detection device 300 may also present information to the subject or record it in the captured image indicating that the imaging range of the gesture detection camera has been changed. Furthermore, the gesture detection device 300 may present information to the subject indicating which of the multiple cameras 100 is the gesture detection camera. The presentation method can be the same as in the second embodiment.

[0074] Furthermore, although the above embodiments described the case where only one gesture control camera is selected, multiple gesture control cameras may be selected. In this case, the gesture information to be detected and the corresponding camera control information may be managed for each gesture control camera, and gesture control may be performed by changing the gesture detection camera according to the imaging state of each gesture control camera.

[0075] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0076] This embodiment includes the following configurations and methods. (Composition 1) A detection means for detecting gesture information of a subject based on an image captured by a second imaging device, which is different from the first imaging device, among a plurality of imaging devices including a first imaging device whose imaging state is controlled according to the gesture of the subject, An acquisition means for acquiring control information for controlling the imaging state of the first imaging device, which is associated with the gesture information detected by the detection means, An information processing apparatus comprising: a control means for controlling the imaging state of the first imaging device based on the control information acquired by the acquisition means. (Configuration 2) The system further includes determination means for determining whether gesture information of the subject can be detected from the image captured by the first imaging device, based on at least one of the image captured by the first imaging device and the imaging state. The detection means is If the determination means determines that the gesture information cannot be detected, the gesture information of the subject is detected based on the image captured by the second imaging device. The information processing device according to configuration 1, characterized in that, if the determination means determines that the gesture information can be detected, the device detects the gesture information of the subject based on the image captured by the first imaging device. (Composition 3) The information processing apparatus according to configuration 1 or 2, further comprising a determination means for determining the second imaging device based on at least one of the imaging state, distribution state, and arrangement state of the plurality of imaging devices. (Composition 4) The aforementioned determination means is The information processing device according to configuration 3, characterized in that, based on the imaging state of the plurality of imaging devices, it determines as the second imaging device one of the plurality of imaging devices: an imaging device that comprehensively images the range in which the subject may move, an imaging device that is imaging the subject, an imaging device that is imaging the range closest to the subject, and an imaging device that is imaging the range of least importance. (Composition 5) The determination means determines, based on the distribution status of the plurality of imaging devices, The information processing device according to configuration 3 or 4, characterized in that any of the plurality of imaging devices that is not currently distributing captured images is selected as the second imaging device. (Composition 6) The determination means determines, based on the arrangement state of the plurality of imaging devices, The information processing device according to any one of configurations 3 to 5, characterized in that, among the plurality of imaging devices, the imaging device located closest to the first imaging device is determined to be the second imaging device. (Composition 7) The information processing device according to any one of configurations 3 to 6, further comprising a changing means for changing the imaging range of the second imaging device determined by the determination means to an imaging range in which gesture information of the subject can be detected from the image captured by the second imaging device. (Composition 8) The information processing device according to configuration 7, characterized in that the modification means presents to the user or records in the captured image information indicating that the imaging range of the second imaging device has been changed. (Composition 9) The system further includes a management means for managing correspondence information that associates the gesture information of the subject with the control information. The information processing apparatus according to any one of configurations 1 to 8, characterized in that the acquisition means refers to correspondence information managed by the management means and acquires the control information associated with the gesture information detected by the detection means. (Composition 10) The information processing device according to configuration 9, characterized in that the management means manages the correspondence information which associates the gesture information of the detection target, determined according to the imaging state of the first imaging device, with the control information. (Composition 11) The information processing apparatus according to any one of configurations 1 to 10, further comprising a presentation means for presenting to the subject information that indicates which of the plurality of imaging devices is capturing an image used for detecting the gesture information by the detection means. (Method 1) The process involves detecting the gesture information of a subject based on an image captured by a second imaging device, which is different from the first imaging device, among a plurality of imaging devices, including a first imaging device whose imaging state is controlled according to the gestures of the subject. A step of acquiring control information for controlling the imaging state of the first imaging device, which is associated with the detected gesture information, An information processing method characterized by comprising the step of controlling the imaging state of the first imaging device based on the acquired control information. [Explanation of symbols]

[0077] 100...Camera, 200...Network, 300...Gesture detection device, 311...Gesture management unit, 312...Image processing unit, 313...Subject detection unit, 314...Gesture detection unit, 315...Camera control unit, 321...Gesture detection feasibility determination unit, 312...Gesture detection camera change unit, 400...Multi-camera control device, 500...Imaging area, 601...Subject, 1000...Imaging system

Claims

1. A detection means for detecting gesture information of a subject based on an image captured by a second imaging device, which is different from the first imaging device, among a plurality of imaging devices including a first imaging device whose imaging state is controlled according to the gesture of the subject, An acquisition means for acquiring control information for controlling the imaging state of the first imaging device, which is associated with the gesture information detected by the detection means, A control means for controlling the imaging state of the first imaging device based on the control information acquired by the acquisition means, A determination means for determining the second imaging device based on at least one of the imaging state, distribution state, and arrangement state of the plurality of imaging devices, A determination means for determining whether gesture information of the subject can be detected from the image captured by the first imaging device, based on at least one of the image captured by the first imaging device and the imaging state, A means for changing the imaging range of the second imaging device to an imaging range that includes the body parts of the subject necessary for detecting the gesture information in the image captured by the second imaging device, The system includes recording means for recording information in the captured image indicating that the imaging range of the second imaging device has been changed by the aforementioned modification means, If the determination means determines that the gesture information cannot be detected, the detection means detects the gesture information of the subject based on the image captured by the second imaging device. If the determination means determines that the gesture information can be detected, the detection means detects the gesture information of the subject based on the image captured by the first imaging device. An information processing device characterized by the following:

2. The aforementioned determination means is The information processing apparatus according to claim 1, characterized in that, based on the imaging state of the plurality of imaging devices, one of the plurality of imaging devices is selected as the second imaging device: an imaging device that comprehensively images the range in which the subject may move, an imaging device that is imaging the subject, an imaging device that is imaging the range closest to the subject, and an imaging device that is imaging the range of least importance.

3. The aforementioned determination means is The information processing device according to claim 1, characterized in that, based on the distribution status of the plurality of imaging devices, one of the plurality of imaging devices that is not currently distributing captured images is determined to be the second imaging device.

4. The aforementioned determination means is The information processing device according to claim 1, characterized in that, based on the arrangement of the plurality of imaging devices, the imaging device located closest to the first imaging device among the plurality of imaging devices is determined to be the second imaging device.

5. The information processing device according to claim 1, characterized in that the modification means presents to the user information indicating that the imaging range of the second imaging device has been changed.

6. The system further comprises a management means for managing correspondence information that associates the gesture information of the subject with the control information. The information processing apparatus according to claim 1, characterized in that the acquisition means refers to correspondence information managed by the management means and acquires the control information associated with the gesture information detected by the detection means.

7. The information processing device according to claim 6, characterized in that the management means manages the correspondence information which associates the gesture information of a detection target determined according to the imaging state of the first imaging device with the control information.

8. The information processing apparatus according to claim 1, further comprising a presentation means for presenting to the subject information that, among the plurality of imaging devices, the imaging device that is capturing the image used for detecting the gesture information by the detection means is presenting to the subject.

9. The information processing apparatus according to claim 1, characterized in that the body part of the subject necessary for detecting the gesture information includes a hand or an arm.

10. The information processing apparatus according to claim 1, characterized in that the body part of the subject necessary for detecting the gesture information includes the feet.

11. A detection step in which, among a plurality of imaging devices including a first imaging device whose imaging state is controlled according to the gestures of the subject, gesture information of the subject is detected based on an image captured by a second imaging device different from the first imaging device, An acquisition step of acquiring control information for controlling the imaging state of the first imaging device, which is associated with the detected gesture information, A control step of controlling the imaging state of the first imaging device based on the acquired control information, A decision step to determine the second imaging device based on at least one of the imaging state, distribution state, and arrangement state of the plurality of imaging devices, A determination step of determining whether gesture information of the subject can be detected from the image captured by the first imaging device, based on at least one of the image captured by the first imaging device and the imaging state, A modification step of changing the imaging range of the second imaging device to an imaging range that includes the body parts of the subject necessary for detecting the gesture information in the image captured by the second imaging device, The system includes a recording step of recording information in the captured image that indicates that the imaging range of the second imaging device has been changed by the modification step, The detection step, if the determination step determines that the gesture information cannot be detected, detects the gesture information of the subject based on the image captured by the second imaging device; if the determination step determines that the gesture information can be detected, detects the gesture information of the subject based on the image captured by the first imaging device. An information processing method characterized by the following:

12. A program for causing a computer to function as each step of the information processing method described in claim 11.

Images