Information processing apparatus, information processing method, and program

Abstract

To facilitate control of an imaging range such that a subject is located at a place desired by a user in the imaging range.SOLUTION: A window used for controlling an image pickup range of image pickup means for picking up an image and including an icon corresponding to a target object serving as a reference for control of the image pickup range included in the window is displayed on display means, a user operation designating a position of the icon in the window displayed on the display means is accepted, and the image pickup range of the image pickup means is controlled in accordance with the position of the icon in the window based on the accepted user operation and a position of the target object detected from the picked up image.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Currently, the video production market is seeing an increase in systems that remotely control network-connected imaging devices. In these systems, there are applications that allow users to change the imaging range (hereinafter referred to as the imaging range) of an imaging device by controlling the PTZ (pan-tilt-zoom) of the imaging device while displaying the image captured by the imaging device.

[0003] Furthermore, one method for controlling the imaging range of an imaging device is to control the PTZ based on the position and size of the subject. Patent Document 1 discloses that in order to capture an image of a subject within a desired imaging range, the ratio between the entire area of ​​the subject and a partial area of ​​the subject (such as the head) is calculated, and the zoom magnification is controlled in accordance with the zoom operation so that the calculated ratio remains constant. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-209620 Summary of the Invention [Problem to be solved by the invention]

[0005] In video production, there are cases where the user controlling the imaging device wants to control the imaging range so that the subject is positioned at a desired location within the imaging range, while being conscious of the composition within the angle of view. In such cases, the user has to control the PTZ using a joystick or the like, while also performing cumbersome operations to control the imaging range so that the subject is positioned at a desired location within the imaging range.

[0006] Therefore, an object of the present invention is to simplify the control of the imaging range so that the subject is positioned in a location desired by the user within the imaging range. [Means for solving the problem]

[0007] In order to solve the above problems, for example, an information processing device of the present invention has the following configuration: That is, it has a display control means for causing a display means to display a window used to control an imaging range of an imaging means that captures an image, the window including an icon corresponding to a target object included in the image and serving as a reference for controlling the imaging range, a reception means for receiving a user operation that specifies a position of the icon in the window displayed on the display means, and a control means for controlling the imaging range of the imaging means in accordance with the position of the icon in the window based on the user operation received by the reception means and the position of the target object detected from the image captured by the imaging means. [Effects of the Invention]

[0008] According to the present invention, it is possible to easily control the imaging range so that the subject is positioned in a location desired by the user within the imaging range. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a system configuration. [Figure 2] FIG. 1 is a diagram illustrating the appearance of an imaging device. [Figure 3] FIG. 2 is a diagram illustrating functional blocks of the imaging device. [Figure 4] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Figure 5] FIG. 10 is a diagram illustrating a GUI for controlling an imaging range. [Figure 6] 10 is a flowchart showing the flow of processing for displaying a frame showing a detection result. [Figure 7] 10 is a flowchart showing the flow of a process for controlling an imaging range. [Figure 8] FIG. 10 is a diagram illustrating a GUI for controlling an imaging range. [Figure 9] FIG. 10 is a diagram illustrating a GUI for specifying a composition for automatic tracking. [Figure 10] 10 is a flowchart showing the flow of automatic tracking processing. [Figure 11] 10 is a flowchart showing the flow of processing for specifying a composition in automatic tracking. [Figure 12] FIG. 2 is a diagram illustrating the hardware configuration of each device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the configurations shown in the drawings.

[0011] (Embodiment 1) 1 is a diagram showing the system configuration of this embodiment. The system of this embodiment includes an imaging device 100, an information processing device 200, a display 400, and a network 300.

[0012] The imaging device 100 and the information processing device 200 are connected to each other via a network 300. The network 300 is realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark).

[0013] The network 300 may be realized by the Internet, a wired local area network (LAN), a wireless LAN, a wide area network (WAN), or the like.

[0014] The imaging device 100 is a device that captures images and functions as an imaging unit capable of controlling the imaging range by driving at least one of pan, tilt, and zoom. The imaging device 100 transmits image data of the captured image, information on the date and time the image was captured, identification information for identifying the imaging device 100, and information on the imaging range of the imaging device 100 to an external device such as an information processing device 200 via a network 300. The information processing device 200 is, for example, a client device such as a personal computer on which a program for implementing the processing functions described below is installed. Note that although the system according to this embodiment uses one imaging device 100, multiple imaging devices 100 may also be used. That is, multiple imaging devices 100 may be connected to the information processing device 200 via the network 300. In this case, the information processing device 200 determines which of the multiple imaging devices 100 captured the transmitted image, for example, using identification information associated with the transmitted image.

[0015] The display 400 is configured with an LCD (Liquid Crystal Display) or the like, and displays images captured by the imaging device 100. The display 400 is connected to the information processing device 200 via a display cable that complies with a communication standard such as HDMI (High Definition Multimedia Interface) (registered trademark). The display 400 and the information processing device 200 may be provided in a single housing.

[0016] Next, an imaging device 100 according to this embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is an example of an external view of the imaging device 100 according to this embodiment. FIG. 3 is an example of functional blocks of the imaging device 100 and an information processing device 200 according to this embodiment. Among the functional blocks of the imaging device 100 shown in FIG. 3, the functions of the image processing unit 112, system control unit 113, pan / tilt / zoom control unit 114, storage unit 115, detection unit 116, communication unit 117, etc. are realized as follows. That is, they are realized by a CPU (Central Processing Unit) 1200 of the imaging device 100 executing a computer program stored in a ROM (Read Only Memory) 1220 of the imaging device 100, which will be described later with reference to FIG. 12.

[0017] The direction in which the optical axis of the lens 101 faces is the imaging direction of the imaging device 100, and a light beam that passes through the lens 101 forms an image on an imaging element of an imaging unit 111 of the imaging device 100. In addition, the lens driving unit 102 is configured with a drive system that drives the lens 101 and changes the focal length of the lens 101. The lens driving unit 102 is controlled by a pan / tilt / zoom control unit 114.

[0018] The pan driving unit 103 is composed of a mechanical driving system that performs panning and a motor that is a driving source, and drives to control rotational driving for rotating the imaging direction of the imaging device 100 in a pan direction 105 (a rotational direction horizontal to the installation surface of the imaging device 100). The pan driving unit 103 is also controlled by a pan / tilt / zoom control unit 114.

[0019] The tilt driving unit 104 is configured with a mechanical drive that performs tilt operation and a motor that is a driving source, and drives to control rotational drive for rotating the imaging direction of the imaging device 100 in a tilt direction 106 (a rotational direction perpendicular to the installation surface of the imaging device 100). The tilt driving unit 104 is controlled by a pan / tilt / zoom control unit 114.

[0020] The imaging unit 111 is configured with an imaging element (not shown), such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The imaging unit 111 photoelectrically converts the subject image formed through the lens 101 to generate an electrical signal. The image processing unit 112 performs image processing such as converting the electrical signal photoelectrically converted by the imaging unit 111 into a digital signal and compression encoding, thereby generating image data.

[0021] The pan-tilt-zoom control unit 114 controls at least one of the pan driving unit 103, the tilt driving unit 104, and the lens driving unit 102 based on instructions transmitted from the system control unit 113. In this way, the imaging device 100 controls the imaging range by controlling at least one of panning, tilting, and zooming.

[0022] The storage unit 115 stores (holds), for example, information indicating an imaging range. The communication unit 117 communicates with the information processing device 200 via an I / F 1240, which will be described later with reference to FIG. 12 . For example, the communication unit 117 transmits image data of an image captured by the imaging device 100 to the information processing device 200 via the network 300. The communication unit 117 also transmits information indicating the imaging range of the imaging device 100. The communication unit 117 also receives control commands, which are commands for controlling the imaging device 100, transmitted from the information processing device 200, and transmits the control commands to the system control unit 113.

[0023] The detection unit 116 executes a process of detecting an object included in an image based on image data generated by the image processing unit 112 (i.e., an image captured by the imaging device 110). Note that the image based on image data here is, for example, an image obtained by decoding image data that has been compression-encoded by the image processing unit 112. Alternatively, for example, the image may be an image before compression-encoding by the image processing unit 112. The detection unit 116 in this embodiment detects a person's face included in the image by performing a process such as pattern matching using a collation pattern (dictionary). Note that, when detecting a person's face from an image, the detection unit 116 may use multiple collation patterns, such as a collation pattern when the person's face is facing forward and a collation pattern when the person's face is facing sideways. In this way, by executing a detection process using multiple collation patterns, improvement in detection accuracy can be expected.

[0024] In this embodiment, a person's face is detected as the subject detected from the image, but the subject is not limited to this and may be, for example, another object such as a car, or another part of a person, such as the upper body or the entire body. The detection unit 116 in this embodiment uses a pattern matching process as a method for detecting an object from an image, but other conventional techniques may also be used to detect an object from an image. In this embodiment, the detection process for detecting an object is described as being performed by the imaging device 110, but the detection process for detecting an object may also be performed by another device, such as the information processing device 200.

[0025] Information on the detection result for an image by the detection unit 116 (hereinafter, "detection result information") is associated with the image and transmitted to the information processing device 100 by the communication unit 117. The detection result information associated with an image includes information on the position and size of the face of a person detected from the image. The position of a person's face on the image is represented by the X and Y coordinates of the center of gravity of a circumscribed rectangle of the person's face on the image, with the upper left vertex of the image being the origin. The size of a person's face on the image is represented by the width and height of the circumscribed rectangle of the person's face in the image. As long as the position and size of the person's face can be identified, the position and size of the person may be represented by other methods. The storage unit 115 stores the detection result information for a person detected by the detection unit 116 in the image in association with the image data of the image.

[0026] The system control unit 113 controls the entire imaging device 100 in accordance with processing executed by the CPU 1200, which will be described later with reference to Fig. 12, and performs the following processing, for example. That is, the system control unit 113 analyzes a control command for controlling the imaging device 100 transmitted from the information processing device 200, and performs processing according to the control command. The system control unit 113 also instructs the pan-tilt-zoom control unit 114 to perform a pan-tilt-zoom operation. When transmitting image data generated by the image processing unit 112 to the information processing device 200, the system control unit 113 also adds information about the imaging time when the image of the image data was captured to the image data.

[0027] The imaging range in this embodiment is determined by the pan value, tilt value, and zoom value of the imaging device 100. The pan value is the angle of the imaging direction (optical axis) in a pan direction 105 of the imaging device 100 when one of the drive ends of the pan driver 103 is set to 0°. The tilt value is the angle of the imaging direction (optical axis) in a tilt direction 106 of the imaging device 100 when one of the drive ends of the tilt driver 104 is set to 0°. The zoom value of the imaging device 100 when an image is captured by the imaging device 100 is calculated from the focal length of the lens 101. The communication unit 117 periodically transmits information on the current pan value, tilt value, and zoom value of the imaging device 100 (hereinafter, PTZ information) to the information processing device 100. When at least one of the pan value, tilt value, and zoom value changes, the communication unit 117 may transmit the current PTZ information (PTZ information after the change) to the information processing device.

[0028] Here, information processing by the information processing device 200 according to this embodiment will be described with reference to the functional blocks of the information processing device 200 shown in Fig. 4. Note that each function of the information processing device 200 is realized as follows using a ROM 1220 and a CPU 1200, which will be described later with reference to Fig. 12. That is, each function shown in Fig. 4 is realized by the CPU 1200 of the information processing device 200 executing a computer program stored in the ROM 1220 of the information processing device 200.

[0029] The communication unit 201 acquires image data transmitted from the imaging device 100 via an I / F 1240, which will be described later with reference to Fig. 12. The recording unit 201 can be realized by a RAM (Random Access Memory) 1210, an HDD (Hard Disk Drive) 1230, etc., which will be described later with reference to Fig. 12, and records information and data related to information processing by the information processing device 200. For example, the recording unit 201 records detection result information and PTZ information transmitted from the imaging device 100 in association with an image.

[0030] The operation receiving unit 203 receives operations performed by the user via input devices (not shown), such as a keyboard or a mouse. The display control unit 204 displays an image based on image data acquired by the communication unit 201 on the display 400. Note that the image based on image data here refers to an image obtained by the information processing device 200 decoding image data that has been compressed, coded, and transmitted. The display control unit 204 displays a GUI (Graphical User Interface) used to control the imaging range of the imaging device 100 on the display 400. A detailed description of the window displayed by the display control unit 204 will be given later.

[0031] The imaging control unit 205 executes processing for controlling the imaging range of the imaging device 100 in accordance with a user operation on a window displayed by the display control unit 204. Specifically, the imaging control unit 207 generates a control command for changing at least one of the pan value, tilt value, and zoom value of the imaging device 100 and transmits the control command to the imaging device 100 via the communication unit 201. The imaging device 100 controls the imaging range in accordance with the transmitted control command. For example, when the operation reception unit 203 receives a user instruction to change the pan value of the imaging device 100 by a predetermined value in the pan direction 105, the imaging control unit 207 executes the following processing. That is, the imaging control unit 205 generates a control command for changing the pan value in the pan direction 105 by the predetermined value and transmits the control command to the imaging device 100 via the communication unit 201. The imaging device 100 changes the pan angle by the predetermined angle in the pan direction 105 by controlling the pan driving unit 103 in accordance with the acquired control command.

[0032] Here, a GUI displayed by the display control unit 204 of the information processing device 200 will be described with reference to FIGS. 5(a) to 5(c). An image captured by the imaging device 100 is displayed on a video display unit 501 in the GUI 500 shown in FIG. 5(a). The image displayed on the video display unit 501 may be a still image, which is one of the images captured by the imaging device 100, or a video consisting of a series of images captured by the imaging device 100. The display control unit 204 superimposes a circumscribing rectangle (in other words, a detection frame) indicating the position and size of a person's face detected from the captured image on the basis of detection result information of the captured image. In the example shown in FIG. 5(a), a circumscribing rectangle 504 is superimposed on the face of person 502, and a circumscribing rectangle 505 is superimposed on the face of person 503. Instead of a circumscribing rectangle, a mark that makes the detected object identifiable, such as an oval or circle, or an arrow, may be superimposed.

[0033] 5A, the GUI 500 displays a PT control button 506, a telephoto button 507, and a wide button 508. Assume that the operation receiving unit 203 receives a user operation to press the directional key indicated by the PT control button 506. At this time, the imaging control unit 205 generates a control command for controlling at least one of the pan and tilt of the imaging device 100 in accordance with the pressed directional key, and transmits the control command to the imaging device 100. Furthermore, when the operation receiving unit 203 receives a user operation to press the telephoto button 507, the imaging control unit 205 generates a control command for increasing the zoom value of the imaging device 100 (zooming in), and transmits the control command to the imaging device 100. Similarly, when the operation receiving unit 203 receives a user operation to press the wide button 508, the imaging control unit 205 generates a control command for decreasing the zoom value of the imaging device 100 (zooming out), and transmits the control command to the imaging device 100. In this way, the user can manually control the imaging range of the imaging device 100 by pressing the PT control button 506, the tele button 507, and the wide button 508. Note that the GUI 500 may also include buttons that can control, for example, the exposure, white balance, or shutter speed of the imaging device 100.

[0034] As shown in FIG. 5A, the GUI 500 also includes a window 509 for controlling the imaging range of the imaging device 100. The window 509 is a GUI for specifying the composition of a subject included in an image and controlling the imaging range of the imaging device 100. The aspect ratio of the rectangle in the window 509 is assumed to be the same as the aspect ratio of an image captured by the imaging device 100. The window 509 also includes an icon 510 corresponding to a target object that serves as a reference for controlling the imaging range. It is assumed that the user has selected the face of a person 502 as the target object that serves as a reference for controlling the imaging range. When the user selects a specific circumscribing rectangle from among the circumscribing rectangles superimposed on the image on the video display unit 501, the imaging control unit 205 identifies the face of the person corresponding to the specific circumscribing rectangle as the target object. At this time, the display control unit 204 changes the display mode of the circumscribing rectangle of the face of the person 502 from that of the circumscribing rectangle corresponding to the face of another person (person 503) so that it is clearly indicated that the person 502 has been selected as the target object. A partial image of the face of the person 502 cut out from the captured image is used as the icon 510 displayed in the window 509. The icon 510 is not limited to this, and may be a graphic that resembles the target object that serves as a reference for controlling the imaging range. The circumscribing rectangle 511 is information indicating the position and size of the icon 510. The position of the icon 510 in the window 509 is indicated by the X and Y coordinates of the center of gravity of the circumscribing rectangle 510, with the upper left vertex of the window 509 as the origin. The size of the icon 510 in the window 509 is indicated by the width and height of the circumscribing rectangle 510. The user can move the position of the icon 510 in the window 509 by dragging the circumscribing rectangle 511, and can change the size of the icon 510 in the window 509 by dragging a vertex of the circumscribing rectangle 511. The GUI 500 shown in Fig. 5(b) shows the state after the position and size of the icon 510 have been changed by the user's operation.

[0035] Assume that the position and size of the icon 510 are changed in accordance with a user operation, and then the execute button 512 is pressed (in the state of FIG. 5(b)). At this time, the imaging control unit 205 executes processing to control the imaging range of the imaging device 100 in accordance with the position of the icon 510 in the window 509 and the position of the target object detected from the image. Specifically, the imaging control unit 205 specifies an imaging range in which the positional relationship (relative position) of the icon 510 with respect to the window 509 is the same as the positional relationship (relative position) of the face of the detected person 504 with respect to the image captured by the imaging device 100. Then, the imaging control unit 205 generates a control command for the specified imaging range. The generated control command is transmitted to the imaging device 100, and the imaging device 100 controls the imaging range in accordance with the transmitted control command. Note that the imaging control unit 205 in this embodiment may further execute processing to control the imaging range based not only on the position of the icon 510 in the window 509 but also on the size of the icon 510 in the window 509. That is, the imaging control unit 205 specifies an imaging range that satisfies the following second condition in addition to the first condition that the positional relationship of the icon 510 with respect to the window 509 is the same as the positional relationship of the face of the detected person 504 with respect to the captured image. The second condition here is that the ratio of the size of the icon 510 with respect to the size of the window 509 is the same as the ratio of the size of the face of the detected person 504 with respect to the size of the captured image. The imaging control unit 205 specifies an imaging range that satisfies the first condition regarding position and the second condition regarding size, and generates a control command to obtain the imaging range. The generated control command is transmitted to the imaging device 100, and the imaging device 100 controls the imaging range in accordance with the control command. FIG. 5(c) shows an image captured after the imaging range has been controlled in this manner. In the image displayed on the video display unit 510 in FIG. 5(c), the positional relationship (relative position) of the face of the person 502 with respect to the image and the positional relationship (relative position) of the icon 510 with respect to the window 509 are the same.5(c), the ratio of the size of the face of person 502 to the size of the image is the same as the ratio of the size of icon 510 to the size of window 509. As described above, according to this embodiment, the user can easily control the imaging range so that the subject is positioned as desired by moving icon 510 to any position in window 509 or changing the size of icon 510.

[0036] Here, a process of displaying an image on which a circumscribing rectangle based on detection result information by the information processing device 200 is superimposed will be described with reference to the flow shown in Fig. 6. Note that the process of the flow shown in Fig. 6 is executed by the functional blocks shown in Fig. 4 which are realized by the CPU 1200 of the information processing device 200 executing a computer program stored in the ROM 1220 of the information processing device 200, for example.

[0037] First, in S601, the communication unit 201 acquires image data of a captured image and detection result information associated with the image. Next, in S602, the display control unit 204 determines whether one or more objects have been detected according to the detection result included in the acquired detection result information. In this embodiment, it is assumed that the determination is made as to whether one or more human faces have been detected. If it is determined that one or more objects have been detected (Yes in S602), the process proceeds to S603. If it is determined that one or more objects have not been detected (No in S602), the process proceeds to S604. In S603, the display control unit 204 displays a circumscribing rectangle determined by information on the position and size of the detected object included in the detection result information on the video display unit 501, superimposing the circumscribing rectangle on the image associated with the detection result information. Furthermore, in S604, the display control unit 204 displays an image based on the acquired image data on the video display unit 501 without superimposing the circumscribing rectangle. In S605, the display control unit 204 determines whether or not to end the processing of the flow shown in Fig. 6, and if it is determined that the processing should be ended (Yes in S605), the processing shown in Fig. 6 is ended. On the other hand, if it is determined that the processing should not be ended (No in S605), the process proceeds to S601, where the communication unit 201 acquires image data of the next image and detection result information associated with that image. Note that the end of the processing shown in Fig. 6 may be instructed by, for example, the user. As described above, by executing the processing of the flow shown in Fig. 6, it is possible to superimpose a circumscribing rectangle on an image and display it on the video display unit 501.

[0038] Next, a control process of the imaging range in response to a user operation of the icon 510 in the window 509 will be described with reference to the process of the flow shown in Fig. 7. Note that the process of the flow shown in Fig. 7 is executed by the functional blocks shown in Fig. 4 which are realized by the CPU 1200 of the information processing device 200 executing a computer program stored in the ROM 1220 of the information processing device 200, for example. Note that the process of the flow shown in Fig. 7 is executed in parallel with the process of the flow shown in Fig. 6.

[0039] First, in S701, the imaging control unit 205 determines whether circumscribing rectangles of one or more objects are displayed superimposed on the image on the video display unit 501. If it is determined that they are not displayed (No in S701), the process proceeds to S707, where the imaging control unit 205 determines whether to end the processing of the flow shown in Fig. 7. If it is determined that the processing is to be ended (Yes in S707), the processing shown in Fig. 7 is ended, and if it is determined that the processing is not to be ended (No in S707), the process proceeds to S701 and the processing of S701 is executed again. If it is determined that circumscribing rectangles of one or more objects are displayed (Yes in S701), the process proceeds to S702. In S702, the imaging control unit 205 determines whether a target object that serves as a reference for controlling the imaging range has been selected by the user. For example, when a user selects a circumscribing rectangle from among the circumscribing rectangles superimposed on the image displayed on the video display unit 501, the imaging control unit 205 identifies the object corresponding to the circumscribing rectangle as the target object. Note that the target object may be selected by, but is not limited to, clicking on the circumscribing rectangle. For example, a number may be assigned to each of the displayed circumscribing rectangles, and when a number is selected from a pull-down menu (not shown) displayed on the GUI 500, the object enclosed by the circumscribing rectangle corresponding to the number may be selected as the target object. FIG. 5A illustrates an example in which the face of a person 502 is identified as the target object. Next, in S703, the operation receiving unit 203 receives a user operation to change the position or size of an icon 510 corresponding to the target object in the window 509. Note that in response to the user operation to change the position or size of the icon 510, the display control unit 204 also changes the display of the icon 510. For example, when an operation is performed to move icon 510 horizontally to the right by a predetermined distance, display control unit 204 accordingly moves icon 510 in window 509 horizontally to the right by a predetermined distance. The state of icon 510 after the position and size of icon 510 have been changed by a user operation on icon 510 is shown in FIG. 5(b).

[0040] Next, in S704, the imaging control unit 205 determines whether the execute button 512 has been pressed by the user. If it is determined that the execute button 512 has not been pressed (No in S704), the process proceeds to S707. If it is determined that the execute button 512 has been pressed (Yes in S704), the process proceeds to S705. In S705, the imaging control unit 205 identifies an imaging range that satisfies a first condition related to the position and a second condition related to the size. Next, in S706, a control command is generated to set the pan, tilt, and zoom values ​​to the identified imaging range, and the communication unit 201 transmits the control command to the imaging device 100. The imaging device 100 controls the imaging range by controlling at least one of pan, tilt, and zoom in accordance with the transmitted control command. An image captured after the imaging range has been controlled in this manner is displayed on the video display unit 501 in FIG. 5(c).

[0041] As described above, according to information processing device 200 of this embodiment, a window used to control the imaging range of imaging device 100, which includes an icon corresponding to a target object that serves as a reference for controlling the imaging range, is displayed on display 400. Then, in response to a user operation to change the position or size of the icon in the window, imaging device 100 is controlled to set the imaging range so that the arrangement of the icon in the window and the arrangement of the target object in a captured image are substantially the same. In this way, it is possible to easily control the imaging range so that the subject is located at a user-desired position in the imaging range.

[0042] In the above description, the pan, tilt, and zoom values ​​of the imaging device 100 are controlled, i.e., the imaging range is controlled by an optical PTZ. However, the imaging range may also be controlled by an electronic PTZ. Electronic PTZ is a function that cuts out a portion of an image captured by the imaging device 100 as the imaging range. The pan / tilt drivable range of the imaging device 100 may be limited by the hardware configuration of the pan driver 103 and tilt driver 104 of the imaging device 100. The pan and tilt drivable range may also be limited by software limitations set by the user in advance. If the pan / tilt of the imaging device 100 is near such a limit, the composition specified in the window 509 may differ from the actual composition of the image captured by the imaging device 100 after the composition specification. Therefore, as shown in FIG. 8 , the window 509 may identifiably display an out-of-range area 801, which indicates an area identified based on the drivable range and in which a target object cannot be placed within the imaging range. The out-of-range area 801 clearly indicates an area where a subject cannot be placed, and the user cannot place the icon 510 in this area. Note that, although the out-of-range area 801 is represented by a black background color in FIG. 8 , this is not limiting and the out-of-range area may be represented by other display modes. Note that, in the example shown in FIG. 8 , an area that cannot be driven in the pan direction 105 is displayed as the out-of-range area 801, but an area that cannot be driven in the tilt direction 106 may also be displayed as the out-of-range area. It is also assumed that there is a limit set to the range in which the zoom value of the imaging device 100 can be changed. In this case, a limit may also be set to the range in which the size of the icon 510 can be changed.

[0043] (Embodiment 2) In this embodiment, a process of controlling the imaging range will be described so that the positional relationship (relative position) of an icon with respect to a window corresponds to the positional relationship (relative position) of a target object with respect to an image captured of the target object to be tracked by controlling the imaging range. In other words, this embodiment is an embodiment that enables specification of the composition of a target object to be tracked by controlling the imaging range. Hereinafter, the process of the control device 110 in embodiment 2 will be described with reference to Figs. 9 to 11.

[0044] The image capturing device 100 of this embodiment is an image capturing device capable of automatic tracking, which controls at least one of panning, tilting, and zooming in accordance with the movement of a target object to be tracked so that the target object always fits within the imaging range. The image capturing device 100 is also capable of receiving a command to execute automatic tracking of the target object (tracking execution command), a command to stop automatic tracking (tracking stop command), and a command to specify the composition of the target object (composition specification command). The composition specification command includes information on the position where the target object to be tracked is to be positioned within the imaging range and its size within the imaging range (hereinafter, "composition information"). The composition information included in the composition specification command may include only information on the position where the target object to be tracked is to be positioned within the imaging range.

[0045] The imaging device 100 of this embodiment will now be described with reference to FIG. 2. The control unit 113 of the imaging device 100 of this embodiment executes automatic tracking of a target object based on composition information. Note that, for example, upon receiving a tracking execution command, the control unit 113 of the imaging device 100 of this embodiment identifies the object closest to the center of the current imaging range as the target object to be tracked, and starts automatic tracking of that object. Note that the condition for determining the target object is not limited to the object closest to the center of the imaging range, and other conditions may also be used. For example, an object that meets either the condition of the largest object in the image or an object having a specific color may be determined as the target object. Furthermore, the conditions here may be set by the user in the information processing device 200.

[0046] Furthermore, the storage unit 115 of the imaging device 100 in this embodiment stores tracking status information indicating the current state of automatic tracking. The tracking status information has three types: "Tracking," "Standing for Tracking (a state in which an automatic tracking execution command has been received but there is no trackable object within the angle of view)," and "Tracking Stopped." Note that, although the types of status include three types in this embodiment, this is not limited to these, and other states may be included. Furthermore, when the tracking status information indicates "Tracking," the tracking status information also stores object identification information that identifies the object being tracked. Furthermore, the storage unit 115 stores composition information included in a composition designation command transmitted from the information processing device 200.

[0047] Next, with reference to Fig. 9, a GUI displayed by the display control unit 204 of the information processing device 200 in this embodiment will be described. In addition to the GUI 500 described with reference to Fig. 5, the GUI 500 shown in Fig. 9 further includes a tracking execution button 901 for instructing execution of automatic tracking, and a stop button 902 for instructing stop of automatic tracking. Here, when the operation reception unit 203 receives a user operation to press the tracking execution button 901, the imaging control unit 205 generates a tracking execution command, and the communication unit 201 transmits the generated tracking execution command to the imaging device 100. Furthermore, when the operation reception unit 203 receives a user operation to press the stop button 902, the imaging control unit 205 generates a tracking stop command, and the communication unit 201 transmits the generated tracking stop command to the imaging device 100. Upon receiving the tracking stop command, the imaging device 100 stores the tracking status information in the storage unit 115 as "tracking stopped".

[0048] 9, it is assumed that icon 510 in window 509 used to control the imaging range is an icon resembling a person. When execute button 512 is pressed, imaging control unit 205 generates a composition designation command including composition information on the position and size of icon 510 in window 509, and communication unit 201 transmits the generated composition designation command to imaging device 100. Then, upon receiving the tracking execution designation command, imaging device 100 performs automatic tracking of the target object in accordance with the composition information included in the composition designation command transmitted from information processing device 200. At this time, system control unit 113 of imaging device 100 refers to the composition information and performs automatic tracking while maintaining an imaging range such that the positional relationship (relative position) of icon 510 with respect to window 509 is the same as the positional relationship (relative position) of the target object with respect to the captured image. In addition to the first condition that the positional relationship (relative position) of icon 510 with respect to window 509 is the same as the positional relationship (relative position) of the target object with respect to the captured image, the imaging range may be controlled to satisfy the following second condition: The second condition here is that the ratio of the size of icon 510 to the size of window 509 is the same as the ratio of the size of the target object to the size of the captured image. System control unit 113 identifies an imaging range that satisfies the first condition regarding position and the second condition regarding size, and performs automatic tracking while controlling the pan value, tilt value, and zoom value to achieve the imaging range.

[0049] Next, the automatic tracking process of the imaging device 100 in this embodiment will be described with reference to the process flow shown in Fig. 10. Note that the process flow shown in Fig. 10 is executed by the functional blocks shown in Fig. 3 which are realized by the CPU 1200 of the imaging device 100 executing a computer program stored in the ROM 1220 of the imaging device 100, for example.

[0050] In S1001, the system control unit 113 acquires a tracking execution command transmitted from the information processing device 200. Next, in S1002, the system control unit 113 refers to the detection result information and determines whether one or more objects have been detected in the most recently captured image. If it is determined that one or more objects have not been detected (No in S1002), the process proceeds to S1006, where the system control unit 113 stores the tracking status information as "tracking standby" in the storage unit 115 and proceeds to S1007. If it is determined that one or more objects have been detected in S1002 (Yes in S1002), the process proceeds to S1003. In S1003, the system control unit 113 refers to the detection result information and determines whether one object has been detected in the most recently captured image. If it is determined that one object has been detected (Yes in S1003), the process proceeds to S1004. In S1004, the system control unit 113 starts automatic tracking of the detected object as the tracking target object, and stores the tracking status information as "tracking" in the storage unit 115. At this time, the tracking status information stored in the storage unit 115 includes object identification information of the identified target object. If it is determined in S1003 that more than one object is being detected (No in S1003), the process proceeds to S1005. In S1005, the system control unit 113 identifies an object that satisfies a predetermined condition as the target object from among the multiple objects currently detected, and starts automatic tracking of the identified target object. At this time, the system control unit 113 also stores the tracking status information as "tracking" in the storage unit 115. At this time, the tracking status information stored in the storage unit 115 includes object identification information of the identified target object. An object that satisfies the predetermined condition is, for example, the object closest to the center of the imaging range, but is not limited to this and may be the largest object in the image, an object having a specific color, or the like.

[0051] In S1007, the system control unit 113 determines whether the current tracking status information is "tracking stopped." If it is determined that it is "tracking stopped" (Yes in S1007), the processing of the flow shown in FIG. 10 ends. On the other hand, if it is determined that it is not "tracking stopped" (No in S1007), the processing proceeds to S1008. Note that when the stop button 902 described with reference to FIG. 9 is pressed by the user, a tracking stop command is transmitted from the information processing device 200 to the image capturing device 100, and upon receiving the tracking stop command, the image capturing device 100 updates the tracking status information to "tracking stopped." In S1008, the system control unit 113 determines whether the current tracking status information is "waiting for tracking." If it is determined that it is "waiting for tracking" (Yes in S1008), the processing proceeds to S1002, and returns to the processing of searching for a target object to be tracked from the detected objects. If it is determined in S1008 that the state is not "waiting for tracking" (No in S1008), the process proceeds to S1009. In S1009, the system control unit 113 refers to the latest detection result information and determines whether the object being tracked is currently being detected. In other words, the system control unit 113 determines whether the object being tracked has been lost. If it is determined that the object being tracked is currently being detected (the object being tracked has not been lost) (Yes in S1009), the process proceeds to S1007, returning to the flow for determining whether "tracking is stopped". If it is determined that the object being tracked is currently not being detected (the object being tracked has been lost) (No in S1009), the process proceeds to S1002, returning to the process of searching for the object being tracked from the detected objects.

[0052] Next, a process for specifying a composition when tracking a target object by the information processing device 200 will be described with reference to Fig. 11. Note that the process of the flow shown in Fig. 11 is executed by, for example, the functional blocks shown in Fig. 4 which are realized by the CPU 1200 of the information processing device 200 executing a computer program stored in the ROM 1220 of the information processing device 200.

[0053] First, in S1001, the operation receiving unit 203 receives a user operation to change the position and size of the icon 510 in the window 509 shown in Fig. 9. Next, in S1002, the imaging control unit 205 determines whether a user operation to press the execute button 512 in Fig. 9 has been received. If it is determined that a user operation to press the execute button has been received (Yes in S1002), the process proceeds to S1003. On the other hand, if it is determined that a user operation to press the execute button has not been received (No in S1002), the process proceeds to S1004. In S1003, the imaging control unit 205 generates a composition designation command including composition information about the position and size of the icon 510 in the window 509, and the communication unit 201 transmits the generated composition designation command to the imaging device 100. Here, if the tracking status information of the imaging device 100 that has acquired the composition specification command is "tracking," the imaging device 100 performs automatic tracking of the target object in accordance with the composition information of the newly acquired composition specification command. Also, if the tracking status information of the imaging device 100 that has acquired the composition specification command is "tracking standby" or "tracking stopped," when it next becomes "tracking," the imaging device 100 performs automatic tracking of the target object in accordance with the composition information of the newly acquired composition specification command. If there is a user instruction to end in S1004 (Yes in S1004), the processing of the flow shown in Fig. 11 ends. If there is no user instruction to end (No in S1004), the process transitions to S1001.

[0054] As described above, when performing automatic tracking, the information processing device 200 in this embodiment uses the composition information of the icon 510 placed by the user in the window 509 as the composition of the object to be tracked. In this way, it becomes possible to perform automatic tracking while maintaining control of the imaging range so that the subject is positioned at a location desired by the user within the imaging range.

[0055] In the above description, an example has been described in which the imaging device 100 controls the imaging range to track the target object based on the transmitted composition information. However, this is not limiting. That is, the imaging control unit 205 of the information processing device 200 may generate a control command for controlling the imaging range to track the target object in accordance with the composition information and transmit the control command to the imaging device 100, thereby controlling the imaging range of the imaging device 100. For example, the imaging control unit 205 executes the following process by referencing information about the position and size of the icon 510 in the window 509, which is included in the composition information. That is, the imaging control unit 205 executes control processing to identify an imaging range that satisfies a first condition related to the position and a second condition related to the size, generate a control command for setting the imaging range to the identified imaging range, and transmit the control command to the imaging device 100. By executing this control processing every time the position or size of the target object detected from the image changes, the information processing device 200 can control the imaging device 100 to automatically track the target object.

[0056] In the above description, an example has been described in which the image capturing device 100 identifies the object closest to the center of the imaging range as the target object to be tracked. However, for example, the target object may be specified by the user. Here, an example of this case will be described. In this case, when the detection unit 116 detects an object from an image, it generates detection result information for the object. At this time, the detection result information associated with the image includes information on the position and size of the object detected in the image, as well as object identification information for identifying the object. Note that, when an object detected in an image of a previous frame is also detected in an image of a current frame, the detection unit 116 assigns the same object identification information to the object in both the previous frame and the current frame. In other words, the same object identification information is assigned to the same object. The detection result information associated with the image and image data of the image are transmitted to the information processing device 200. Then, when an object is selected as the target object on the GUI 500 of the information processing device 200 and an instruction to execute tracking is given, a tracking execution command including object identification information for identifying the object is transmitted to the image capturing device 100. Based on the object identification information included in the received tracking execution command, the system control unit 113 identifies the target object from among the objects currently detected by the detection unit 116. Then, the system control unit 113 performs automatic tracking while controlling the imaging range so that the target object has the specified composition, in accordance with the composition information included in the further received composition designation command. In this way, the user may designate the target object to be tracked.

[0057] (Other embodiments) Next, the hardware configuration of the information processing device 200 for realizing each function of the above-described embodiment will be described with reference to Fig. 12. Note that, although the hardware configuration of the information processing device 200 will be described in the following explanation, it is assumed that the imaging device 100 is also realized by a similar hardware configuration.

[0058] The information processing device 200 in this embodiment includes a CPU 1200 , a RAM 1210 , a ROM 1220 , an HDD 1230 , and an I / F 1240 .

[0059] The CPU 1200 is a central processing unit that controls the information processing device 110. The RAM 1210 temporarily stores computer programs executed by the CPU 1200. The RAM 1210 also provides a work area used by the CPU 1200 when executing processing. The RAM 1210 also functions as, for example, a frame memory or a buffer memory.

[0060] The ROM 1220 stores programs and the like that are used by the CPU 1200 to control the information processing device 110. The HDD 1230 is a storage device that records image data and the like.

[0061] The I / F 1210 communicates with external devices via the network 300 in accordance with TCP / IP, HTTP, or the like.

[0062] Although the above-described embodiments have been described with reference to examples in which the CPU 1200 executes the processing, at least a portion of the processing by the CPU 1200 may be performed by dedicated hardware. For example, the processing of displaying a GUI (Graphical User Interface) or image data on the display 400 may be performed by a GPU (Graphics Processing Unit). Furthermore, the processing of reading program code from the ROM 1220 and loading it into the RAM 1210 may be performed by a DMA (Direct Memory Access) that functions as a transfer device.

[0063] The present invention can also be realized by a process in which one or more processors read and execute a program that realizes one or more functions of the above-described embodiments. The program may be supplied to a system or device having a processor via a network or a storage medium. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions of the above-described embodiments. Each unit of the information processing device 200 may be realized by hardware shown in FIG. 12 or by software. One or more functions of the information processing device 200 according to the above-described embodiment may be implemented by another device.

[0064] Although the present invention has been described above with reference to the embodiments, the above embodiments merely illustrate specific examples of how the present invention can be implemented, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from the technical concept or main features of the present invention. For example, combinations of the embodiments are also included in the disclosure of this specification. [Explanation of symbols]

[0065] 100 Imaging device 200 Information processing device 300 Network 400 displays 201 Communications Department 202 Recording Department 203 Operation reception section 204 Display control unit 205 Imaging control unit

Claims

[Claim 1] A window used to control an imaging range of an imaging means for capturing an image, a display control means for displaying on a display means a window including an icon corresponding to a target object included in the image and serving as a reference for controlling the imaging range; a receiving means for receiving a user operation for specifying a position of the icon in the window displayed on the display means; an information processing device comprising: a control means for controlling an imaging range of the imaging means in accordance with a position of the icon in the window based on a user operation accepted by the accepting means and a position of the target object detected from an image taken by the imaging means.

Images