Control device, control method, and program

Abstract

To facilitate control of an imaging range to locate a subject at a place desired by a user in the imaging range.SOLUTION: An information processing apparatus acquires an image picked up by imaging means, causes display means to display a plurality of arrangement candidates including at least a first arrangement candidate corresponding to a composition where a subject in an imaging range of the imaging means is arranged at a first position and a second arrangement candidate corresponding to a composition where the subject in the imaging range is arranged at a second position different from the first position, and when one of the plurality of arrangement candidates is selected by a user, executes processing of controlling the imaging range based on information on the selected arrangement candidate. The plurality of arrangement candidates displayed by the display means is determined according to the position of the subject or the direction of the face of the subject detected from the image.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a method for controlling an imaging device.

Background Art

[0002] Currently, in the video production market, systems for remotely controlling imaging devices connected to a network are increasing. In such a system, there is an application in which, while displaying the video captured by the imaging device, the range captured by the imaging device (hereinafter, the imaging range) can be changed by controlling the PTZ (pan, tilt, zoom) of the imaging device. A user who uses the application can change the imaging range using a device such as a joystick.

[0003] Also, as a method for controlling the imaging range of an imaging device, there is a method of controlling the PTZ based on the position and size of a subject. In Patent Document 1, in order to photograph a subject within a desired shooting range, the ratio of two regions, namely, the entire region of the subject and a partial region of the subject (such as the head), is calculated, and the zoom magnification is controlled so that the calculated ratio becomes constant in accordance with a zoom operation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a video production site, there are cases where a cameraman wants to control the imaging range so that a subject is located at a desired position within the imaging range while being conscious of the composition within the angle of view. Here, the operation of controlling the imaging range in order to obtain a composition in which the subject is located at a desired position within the imaging range while the user controls the PTZ using a joystick is complicated.

[0006] Therefore, the present invention aims to simplify the control of the imaging range so that the subject is located in a position desired by the user within the imaging range. [Means for solving the problem]

[0007] To solve the above problems, for example, the information processing apparatus according to the present invention has the following configuration: an acquisition means for acquiring an image captured by an imaging means; a display control means for displaying a plurality of arrangement candidates on a display means, which include at least a first arrangement candidate corresponding to a composition in which the subject is placed at a first position within the imaging range of the imaging means, and a second arrangement candidate corresponding to a composition in which the subject is placed at a second position different from the first position within the imaging range; and a control means that, when one of the plurality of arrangement candidates is selected by a user, executes a process to control the imaging range by controlling the imaging direction of the imaging means based on the position of the subject in the composition corresponding to the selected arrangement candidate and the position of the subject detected from the image. Processing means for performing grouping of multiple people detected in the aforementioned image, The plurality of arrangement candidates displayed on the display means are determined according to the position of the subject detected from the image or the orientation of the subject's face. The display control means displays on the display means one or more arrangement candidates for the first group grouped by the processing means, and one or more arrangement candidates for the second group grouped by the processing means, as the plurality of arrangement candidates. ru. [Effects of the Invention]

[0008] According to the present invention, it is possible to easily control the imaging range so that the subject is located in a position desired by the user within the imaging range. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the system. [Figure 2] This figure shows an example of the external appearance of an imaging device. [Figure 3] This diagram shows the functional block of the imaging device. [Figure 4] This diagram shows the functional blocks of an information processing device. [Figure 5]It is a diagram for explaining a GUI displayed on a display. [Figure 6] It is a diagram for explaining placement candidates. [Figure 7] It is a flowchart showing the flow of a process for displaying placement candidates. [Figure 8] It is a diagram for explaining detection result information. [Figure 9] It is a diagram for explaining a GUI displayed on a display. [Figure 10] It is a diagram for explaining a GUI displayed on a display. [Figure 11] It is a diagram for explaining a GUI displayed on a display. [Figure 12] It is a diagram for explaining a GUI displayed on a display. [Figure 13] It is a flowchart showing the flow of a process for displaying placement candidates. [Figure 14] It is a diagram showing an example of the hardware configuration of each device.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the illustrated configurations.

[0011] (Embodiment 1) FIG. 1 is a diagram showing the system configuration in the present embodiment. The system in the present embodiment includes an imaging device 100, an information processing device 200, a display 400, and a network 300.

[0012] The imaging device 100 and the information processing device 200 are connected to each other via the network 300. The network 300 is realized from a plurality of routers, switches, cables, etc. conforming to a communication standard such as ETHERNET (registered trademark).

[0013] Note that the network 300 may be implemented by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless Lan), a WAN (Wide Area Network), or the like.

[0014] The imaging device 100 is a device that captures images and functions as imaging means capable of controlling the imaging range. The imaging device 100 transmits, via the network 300, the image data of the captured image, the information of the imaging date and time when the image was captured, the identification information for identifying the imaging device 100, and the information of the imaging range of the imaging device 110 to an external device such as the information processing device 200. The information processing device 200 is, for example, a client device such as a personal computer in which a program for realizing the functions of the processes described later is installed. In the system according to the present embodiment, the imaging device 100 is assumed to be one, but may be plural. That is, a plurality of imaging devices 100 may be connected to the information processing device 200 via the network 300. In this case, the information processing device 200 determines, for example, which imaging device 100 among the plurality of imaging devices 100 captured the transmitted image using the identification information associated with the transmitted image.

[0015] The display 400 is composed of an LCD (Liquid Crystal Display) or the like and displays an image captured by the imaging device 100 or the like. The display 400 is connected to the information processing device 200 via a display cable conforming to a communication standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Note that the display 400 and the information processing device 200 may be provided in a single housing.

[0016] Next, the imaging device 100 according to this embodiment will be described with reference to Figures 2 and 3. Figure 2 is an example of an external view of the imaging device 100 according to this embodiment. Figure 3 is an example of a functional block of the imaging device 100 and information processing device 200 according to this embodiment. Of the functional blocks of the imaging device 100 shown in Figure 3, the functions of the image processing unit 112, system control unit 113, pan-tilt-zoom control unit 114, storage unit 115, communication unit 116, etc., are realized as follows. That is, they are realized by the CPU (Central Processing Unit) 1400 of the imaging device 100 executing a computer program stored in the ROM (Read Only Memory) 1420 of the imaging device 100, which will be described later with reference to Figure 14.

[0017] The optical axis of lens 101 points in the same direction as the imaging direction of the imaging device 100, and the light beam passing through lens 101 forms an image on the image sensor of the imaging unit 111 of the imaging device 100. The lens drive unit 102 is composed of a drive system that drives lens 101 and changes the focal length of lens 101. The lens drive unit 102 is controlled by the pan-tilt-zoom control unit 114.

[0018] The pan drive unit 103 consists of a mechanical drive system and a motor that acts as a drive source for panning, and controls the rotational drive to rotate the imaging device 100 in the pan direction 105. The pan drive unit 103 is controlled by the pan-tilt-zoom control unit 114.

[0019] The tilt drive unit 104 consists of a mechanical drive and a motor that acts as a drive source for the tilt operation, and controls the rotational drive to rotate the imaging device 100 in the tilt direction 106. The tilt drive unit 104 is controlled by the pan-tilt-zoom control unit 114.

[0020] The imaging unit 111 is composed of an image sensor (not shown), such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The imaging unit 111 then converts the image of the subject formed through the lens 101 into an electrical signal using photoelectric conversion. The image processing unit 112 performs image processing, such as converting the electrical signal converted into a digital signal by the imaging unit 111 into a digital signal and compression encoding, to generate image data.

[0021] The pan-tilt-zoom control unit 114 controls the pan, tilt, and zoom of the imaging device 100 by controlling the pan drive unit 103, the tilt drive unit 104, and the lens drive unit 102 based on instructions transmitted from the system control unit 113.

[0022] The storage unit 115 stores (holds) information indicating the imaging range, for example. The communication unit 116 communicates with the information processing device 200 via the I / F 1440, which will be described later with reference to Figure 14. For example, the communication unit 116 transmits image data of the image captured by the imaging device 100 to the information processing device 200 via the network 300. The communication unit 116 also transmits information indicating the imaging range of the imaging device 100. Furthermore, the communication unit 116 receives control commands, which are commands for controlling the imaging device 100, transmitted from the information processing device 200, and transmits them to the system control unit 113.

[0023] The system control unit 113 controls the entire imaging device 100 according to the processing performed by the CPU 1400, as described later with reference to Figure 14, and performs the following processing, for example: The system control unit 113 analyzes the control commands for controlling the imaging device 100 transmitted from the information processing device 200 and performs processing according to the control commands. The system control unit 113 also instructs the pan-tilt-zoom control unit 114 to perform pan-tilt-zoom operations. Furthermore, when the system control unit 113 transmits the image data generated by the image processing unit 112 to the information processing device 200, it also adds information about the time the image data was captured to the image data.

[0024] In this embodiment, the imaging range is determined by the pan value, tilt value, and zoom value of the imaging device 100. The pan value is the angle of the imaging direction (optical axis) in the pan direction 105 of the imaging device 100, with one of the drive ends of the pan drive unit 103 set to 0°. The tilt value is the angle of the imaging direction (optical axis) in the tilt direction 106 of the imaging device 100, with one of the drive ends of the tilt drive unit 104 set to 0°. The zoom value of the imaging device 100 when an image is captured by the imaging device 100 is calculated from the focal length of the lens 101.

[0025] Here, the information processing of the information processing device 200 according to this embodiment will be described with reference to the functional blocks of the information processing device 200 shown in Figure 4. Each function of the information processing device 200 is implemented as follows, using the ROM 1420 and CPU 1400, which will be described later with reference to Figure 14. That is, each function shown in Figure 4 is implemented by the CPU 1400 of the information processing device 200 executing a computer program stored in the ROM 1420 of the information processing device 200.

[0026] The communication unit 201 acquires image data transmitted from the imaging device 100 via the I / F 1440, which will be described later with reference to Figure 14. The recording unit 202 can be implemented by a RAM (Random Access Memory) 1010 or an HDD (Hard Disk Drive) 1030, which will be described later with reference to Figure 10, and records information and data related to information processing by the information processing device 200. For example, the recording unit 202 records information regarding the position of a subject detected from the image. The recording unit 202 also records controllability information, which indicates which of the pan value, tilt value, and zoom value the imaging device 100 can control.

[0027] The detection unit 203 performs a detection process to detect subjects in the image acquired by the communication unit 201. In this embodiment, the detection unit 203 uses a matching pattern (dictionary) to perform processing such as pattern matching to detect a person as a subject in the image. In addition, when detecting a person from an image, multiple matching patterns may be used, such as a matching pattern for when the person is facing forward and a matching pattern for when the person is facing sideways. By performing a detection process using multiple matching patterns in this way, an improvement in detection accuracy can be expected.

[0028] Furthermore, matching patterns may be prepared for viewing a person from other angles, such as from an oblique direction or from above. Also, when detecting a person from an image, it is not always necessary to prepare matching patterns (dictionaries) that show the features of the entire body; matching patterns may be prepared for parts of the person, such as the upper body, lower body, head, face, or feet.

[0029] In this embodiment, a person is detected as the subject detected from the image, but it is not limited to a person; for example, it could be another object such as a car. Also, in this embodiment, the detection unit 203 uses pattern matching processing as a method for detecting a specific object from an image, but other prior arts may be used to detect a specific object from an image.

[0030] The identification unit 204 identifies subjects included in the image as a reference for controlling the imaging range of the imaging device 100. The imaging range of the imaging device 100 is controlled based on the subjects (people) in the image identified by the identification unit 204.

[0031] The operation reception unit 205 receives operations performed by the user via an input device such as a keyboard or mouse (not shown). The display control unit 206 displays the image acquired by the communication unit 201 on the display 400. In this embodiment, the display control unit 206 also displays on the display 400 multiple arrangement candidates corresponding to the composition in which the subject is placed within the imaging range. A detailed explanation of the arrangement candidates displayed by the display control unit 206 will be given later.

[0032] The range control unit 207 executes a process to control the imaging range of the imaging device 100. Specifically, the range control unit 207 generates a control command to change at least one of the pan value, tilt value, and zoom value of the imaging device 100 and transmits it to the imaging device 100 via the communication unit 201. The imaging device 100 controls the imaging range according to the transmitted control command. For example, if the operation reception unit 205 receives a user instruction to change the pan value of the imaging device 100 by a predetermined amount in the pan direction 105, the range control unit 207 executes the following process: That is, the range control unit 207 generates a control command to change the pan value by a predetermined amount in the pan direction 105 and transmits it to the imaging device 100 via the communication unit 201. The imaging device 100 changes the pan angle by a predetermined angle in the pan direction 105 by controlling the pan drive unit 103 according to the acquired control command.

[0033] Next, with reference to Figures 5(a) and 5(b), the information on the placement candidates displayed on the display 400 by the display control unit 206 will be explained. The display window 501 shown in Figure 5(a) is displayed on the display 400 by the display control unit 206 and includes an image display area 502 on which the image captured by the imaging device 100 is displayed. The display window 501 also includes multiple placement candidates 505 to 510. As shown in Figure 5(a), an image including a person 503 identified as a reference subject by the identification unit 204 is displayed in the image display area 502. At this time, the display control unit 206 displays a frame 504 superimposed on the image corresponding to the position of the person 503, based on the detection result information from the detection unit 203. Multiple placement candidates 505 to 510 are displayed, including at least a placement candidate 505 corresponding to a composition in which the subject is placed at a first position within the imaging range of the imaging device 100, and a placement candidate 506 corresponding to a composition in which the subject is placed at a second position different from the first position within the imaging range. As shown in Figure 5(a), the display window 501 shows both the captured image and multiple placement candidates 505 to 510. Placement candidate 505 is displayed as a figure with the subject icon 511 positioned on the left within the rectangular area indicating the imaging range. Placement candidate 506 is displayed as a figure with the subject icon 512 positioned in the center within the rectangular area indicating the imaging range. Placement candidate 507 is displayed as a figure with the subject icon 513 positioned on the right within the rectangular area indicating the imaging range. Placement candidate 508 is displayed as a figure with the subject icon 514 positioned on the left and larger within the rectangular area indicating the imaging range. Placement candidate 509 is displayed as a figure with the subject icon 515 positioned in the center and larger within the rectangular area indicating the imaging range. Placement candidate 510 is displayed as a figure with the subject icon 516 positioned on the right and larger within the rectangular area indicating the imaging range. Each rectangular area of ​​the placement candidate corresponds to the imaging range of the imaging device 100, and the aspect ratio of the rectangular area is the same as the aspect ratio of the imaging range and the captured image.

[0034] Here, we will explain the case where placement candidate 510 is selected from among the multiple placement candidates displayed in the display window 501 of Figure 5(a). When placement candidate 510 is selected by user operation, the range control unit 207 performs the following processing. That is, the range control unit 207 identifies the relative position of the centroid of icon 516 with respect to the rectangular area of ​​placement candidate 510 (hereinafter referred to as the relative position) and the relative position of the centroid of person 503, which is the reference subject, with respect to the imaging range of the current imaging device 100. Then, the range control unit 207 determines the pan value and tilt value so that the relative position of icon 516 with respect to the rectangular area of ​​placement candidate 510 and the relative position of person 503 with respect to the imaging range are approximately the same. Furthermore, the range control unit 207 determines the zoom value so that the relative size of icon 511 with respect to the size of the rectangular area of ​​placement candidate 510 and the relative size of person 503 with respect to the size of the actual imaging range are approximately the same. The range control unit 207 then generates control commands to control the pan, tilt, and zoom values ​​determined by the unit, and transmits them to the imaging device 100 via the communication unit 201. The imaging device 100 controls the imaging range to match the pan, tilt, and zoom values ​​specified in the transmitted control commands. After the imaging range of the imaging device 100 is controlled in this way, the captured image is transmitted from the imaging device 100 to the information processing device 200. Figure 5(b) shows an example in which the image captured and transmitted after the imaging range of the imaging device 100 is controlled according to the selection of placement candidates is displayed in the image display area 502 by the display control unit 206. In this case, as shown in Figure 5(b), an image is obtained in which the positional relationship and size of the icon 516 in the rectangular area of ​​the placement candidate 510 are the same as the positional relationship and size of the subject (person 503) relative to the actual imaging range. Furthermore, as shown in Figure 5(b), the display control unit 206 makes the display mode of placement candidate 510 different from that of the other placement candidates 505 to 509, depending on the selection of placement candidate 510. In this way, it is possible to inform the user that placement candidate 510 has been selected.

[0035] In addition, while the example shown in Figure 5 described the case of a single-person arrangement, here we will explain the case of arrangements for other numbers of people with reference to Figure 6. Figure 6(a) shows examples of multiple arrangement candidates 601 to 606 for one person, each of which has been created in advance and stored in the memory unit 202. Figure 6(b) shows examples of multiple arrangement candidates 607 to 617 for two people, each of which has been created in advance and stored in the memory unit 202. Here, arrangement candidates 607 to 611 shown in Figure 6(b) are arrangement candidates when the two people are positioned side by side. Arrangement candidates 612 to 614 are arrangement candidates when the two people are positioned diagonally upwards to the right. Arrangement candidates 615 to 617 are arrangement candidates when the two people are positioned diagonally downwards to the right. Thus, for two-person arrangement candidates, at least one or more arrangement candidates for the case where the two people are in a first positional relationship, and one or more arrangement candidates for the case where the two people are in a second positional relationship different from the first positional relationship, may be included. Furthermore, when there are two reference subjects in the current field of view, the display control unit 206 may change the arrangement candidates to display according to the positional relationship of each reference subject. For example, if there are two reference subjects in the current field of view and the people are in a side-by-side positional relationship, the display control unit 206 may display arrangement candidates 607 to 611 corresponding to the side-by-side positional relationship. Also, arrangement candidate 709 shown in Figure 6(c) represents arrangement candidate 618 for three people. In the example shown in Figure 6(c), when arrangement candidate 618 for three people is selected, the imaging direction is controlled so that each person is included in the field of view. However, even for arrangement candidates for three people, one or more arrangement candidates may be prepared for each positional relationship of each person, similar to arrangement candidates for two people. Note that the layout options to be displayed are not limited to those shown in Figures 6(a) to (c), and may include other layout options as well.

[0036] Figure 7 is a flowchart showing the process of displaying the composition pattern in this embodiment. The processing shown in Figure 7 is executed by a functional block of the information processing device 200 shown in Figure 4, which is realized by the CPU 1400 of the imaging device 100 executing a computer program stored in the ROM 1420 of the information processing device 200.

[0037] First, in step S701, the display control unit 206 determines whether a subject (person) has been detected by the detection unit 203 from the captured image and whether that subject (person) has been identified as a reference subject. The identification unit 204 may identify all subjects detected by the detection unit 203 as reference subjects, or it may identify a subject selected by the user from among the subjects detected by the detection unit 203 as a reference subject. If a reference subject is identified from the image (Yes in S701), the process proceeds to S702; otherwise, the process in S701 is repeated.

[0038] Next, in S702, the display control unit 206 determines whether there has been a change in the number or positional relationship of the reference subjects in the image. If it determines that there has been a change (Yes in S702), it proceeds to S703; if it determines that there has been no change (No in S702), it proceeds to S705. For example, if the number of reference subjects changes from one to two, it is determined that there has been a change. Also, for example, if the number of reference subjects in the image is two, and their positional relationship changes from being lined up diagonally to the upper right to being lined up side by side, it is determined that there has been a change.

[0039] In S703, the display control unit 206 determines one or more placement candidates according to the current reference subject and its positional relationship. For example, if there is one reference subject, the display control unit 206 determines the placement candidates 601 to 606 stored in the memory unit 202 to be displayed. If there are two reference subjects and they are positioned side by side, the display control unit 206 determines the placement candidates 607 to 611.

[0040] In S704, the display control unit 206 updates the GUI 501 so that one or more of the placement candidates determined in S703 are displayed on the GUI 501.

[0041] Next, in S705, the display control unit 206 determines whether there is a termination instruction. If it is determined that there is a termination instruction (Yes in S705), it terminates the processing of the flow shown in Figure 7. If it is determined that there is no termination instruction (No in S705), it proceeds to S701.

[0042] In the above example, we have described cases in which the identification unit 204 uniformly identifies all subjects detected from the image as reference subjects, or identifies subjects selected by the user from among the subjects detected from the image as reference subjects, but it is not limited to these cases. For example, people in the image may be prioritized, and reference subjects may be identified based on the results. Here, the detection result information 800 shown in Figure 8 is information generated by the detection unit 203 based on the subject detection result of the detection unit 203 for the image, and is stored in the storage unit 202. The detection result information 800 stores a detected person ID 801, which is an ID that identifies the detected person, coordinates 802 that indicate the position of the person, a size 803 that indicates the size of the person, and a stay time 804 that indicates the time the person is continuously visible in the image. The coordinates 802 indicate the position of the person detected from the latest image, and are the XY coordinates of the centroid of the person when the upper left vertex of the image is taken as the origin. The size 803 indicates the vertical size of the person detected from the latest image in that image. Here, the identification unit 204 may prioritize the people in the image in order of the length of their stay time 804, and identify the person with the highest priority as the reference subject. For example, the person corresponding to the detected person ID 801 with the longest stay time 804 may be identified as the reference subject. Alternatively, the number of placement candidate patterns that are preferentially displayed for people with a long stay time 804 may be increased.

[0043] As described above, the information processing device 200 in this embodiment displays a plurality of different arrangement candidates on the GUI used to control the imaging range, depending on the number or positional relationship of reference subjects in the image. The information processing device 200 then controls the imaging range of the imaging device 100 so that an image with a composition of subjects similar to that of the arrangement candidate is captured, according to the arrangement candidate selected from the plurality of displayed arrangement candidates. In this way, it is possible to easily control the imaging range so that the subjects are located in the location desired by the user within the imaging range.

[0044] (Embodiment 2) In Embodiment 2, multiple reference subjects in the image are grouped, and placement candidates corresponding to each of the multiple groups are displayed. Note that only the differences from Embodiment 1 will be described, and the same or equivalent components and processes as in Embodiment 1 will be denoted by the same reference numerals, and redundant explanations will be omitted.

[0045] The display window 501 shown in Figure 9 is displayed on the display 400 by the display control unit 206 in this embodiment. The display window 501 in Figure 9 includes an image display area 502 on which an image captured by the imaging device 100 is displayed. In the example shown in Figure 9, the displayed image includes four people 901 to 904. In this embodiment, the identification unit 204 identifies the people 901 to 904 detected by the detection unit 203 as reference subjects and groups the people 901 to 904. For example, the identification unit 204 groups people within a predetermined distance of each person as the same group based on their relative positions. In the example in Figure 9, the identification unit 204 groups people 901 to 902 as the first group and people 903 to 904 as the second group. Then, a detection frame 905 is superimposed on the image for the first group and a detection frame 906 for the second group to indicate the position of each group.

[0046] In the example shown in Figure 9, the display control unit 206 in this embodiment determines the display candidates for each grouped group based on the number of people and their relative positions within that group. The process of determining the display candidates is performed in the process S704 shown in Figure 7. In Figure 9, the display control unit 206 displays the placement candidates 907 to 909, which correspond to the horizontal positional relationship between person 901 and person 902, as placement candidates corresponding to the first group. At this time, the display manner of the detection frame 905 for the first group and the display manner of the frames for placement candidates 907 to 909 are made similar so that the user can identify that the first group and placement candidates 907 to 909 are associated. In Figure 9, the detection frame 905 for the first group and the frames for placement candidates 907 to 909 are displayed with solid lines. The display control unit 206 also displays the placement candidates 910 to 912, which correspond to the horizontal positional relationship between person 903 and person 904, as placement candidates corresponding to the second group. At this time, the display manner of the detection frame 906 for the first group and the display manner of the frames for placement candidates 910 to 912 are made similar so that the user can identify that the second group is associated with placement candidates 910 to 912. In Figure 9, the detection frame 906 for the second group and the frames for placement candidates 910 to 912 are displayed with dotted lines. It is also possible to set an upper limit on the number of people that can be included in a single group.

[0047] Here, we will explain an example where placement candidate 911 is selected from the placement candidates 907 to 912 shown in Figure 9. In this case, the range control unit 207 performs processing to control the imaging range based on placement candidate 911 corresponding to the second group and the positions of people 903 and 906 belonging to the second group. Specifically, the range control unit 207 performs the following processing: It determines pan and tilt values ​​such that the relative positions of the centroids of icons 913 and 914 with respect to the rectangular area of ​​placement candidate 911 are approximately the same as the relative positions of the centroids of people 903 and 906 with respect to the current imaging range of the imaging device 100. The range control unit 207 also determines zoom values ​​such that the ratio of the total size of icons 914 to 914 with respect to the size of the rectangular area of ​​placement candidate 911 is approximately the same as the ratio of the total size of people 903 to 904 with respect to the size of the imaging range. The range control unit 207 then generates control commands that result in the determined pan, tilt, and zoom values, and transmits them to the imaging device 100 via the communication unit 201. The imaging device 100 controls the imaging range to match the pan, tilt, and zoom values ​​specified in the transmitted control commands.

[0048] In this embodiment, an example of providing placement candidates for each grouped group has been described, but further placement candidates may be provided for the case where all reference subjects in the image are grouped together. The display window 501 in this case will be described with reference to Figure 10. The display window 501 shown in Figure 10 is the same as the display window 501 shown in Figure 9, but with placement candidate 907 removed, and includes a frame 1001 corresponding to the group when all reference subjects in the image are grouped together, and a placement candidate 1002 corresponding to that group. When placement candidate 1001 is selected by the user, the range control unit 207 performs, for example, the following processing. That is, it determines pan and tilt values ​​such that the relative position of the centroid position of all icons in placement candidate 1002 with respect to the rectangular area of ​​placement candidate 1002 and the relative position of the centroid position of all reference subjects in the image with respect to the imaging range are approximately the same. The range control unit 207 also determines the zoom value by performing, for example, the following processing. In other words, the zoom value is determined such that the ratio of the total size of all icons in placement candidate 1002 to the size of the rectangular area of ​​placement candidate 1002 is approximately the same as the ratio of the total size of all reference subjects in the imaging range to the size of the imaging range. The range control unit 207 then generates control commands that result in the determined pan, tilt, and zoom values ​​and transmits them to the imaging device 100 via the communication unit 201. The imaging device 100 controls the imaging range to match the pan, tilt, and zoom values ​​instructed in the transmitted control commands. Further placement candidates may be provided in this manner, where all reference subjects in the image are grouped into a single group.

[0049] As described above, the information processing device 200 in this embodiment groups people in an image and displays placement candidates corresponding to each group. In this way, it is possible to easily control the imaging range so that the subject is located in the location desired by the user within the imaging range.

[0050] (Embodiment 3) Embodiment 3 describes an example of displaying placement candidates based on the orientation of the face of a reference subject in an image. Note that this description will primarily focus on the differences from Embodiment 1, using the same reference numerals for identical or equivalent components and processes as in Embodiment 1, and omitting redundant explanations.

[0051] Figure 11 shows the display window 501 in this embodiment, which is displayed on the display 400 by the display control unit 206. In the display window 501 shown in Figure 11, the image displayed in the image display area 502 includes person 1101 and person 1102, which have been identified as reference subjects. A detection frame 1103 is superimposed on the position of person 1101, and a detection frame 1104 is superimposed on the position of person 1102. The display window 501 in Figure 11 also includes placement candidates 1105 to 1107 corresponding to person 1101 and placement candidates 1108 to 1110 corresponding to person 1102. In this embodiment, the detection unit 203 also detects the face orientation of each of the people 1101 to 1102 in the image, and the display control unit 206 determines one or more placement candidates to display based on the face orientation of the people. Specifically, for person 1101 facing right in the image, the display control unit 206 displays placement candidates 1105 to 1107 as placement candidates associated with person 1101, such that space is created in the direction that person 1101 is facing. As shown in Figure 11, placement candidates 1105 to 1107 corresponding to person 1101 are placement candidates where the space on the right side of the icon is larger than the space on the left side. Similarly, for person 1102 facing left in the image, the display control unit 206 displays placement candidates 1108 to 1110 as placement candidates associated with person 1102, such that space is created in the direction that person 1102 is facing. As shown in Figure 11, placement candidates 1108 to 1110 corresponding to person 1102 are placement candidates where the space on the right side of the icon is larger than the space on the left side.

[0052] As described above, the information processing device 200 in this embodiment displays one or more placement candidates for each reference subject in the image, based on the orientation of the reference subject's face. This makes it easy to control the imaging range so that the subject is located in the location desired by the user within the imaging range. Furthermore, by displaying placement candidates that create space in the direction the reference subject is facing, it becomes possible to control the imaging range to one that allows the user to understand where the reference subject is facing, depending on the selection of the placement candidate.

[0053] (Embodiment 4) Embodiment 4 of the present invention will now be described. Based on the current pan, tilt, and zoom values ​​of the imaging device 100, the information processing device 200 of this embodiment identifies placement candidates that, even if selected by the user, cannot be controlled due to the pan-tilt-zoom drive limits (hereinafter referred to as "uncontrollable candidates") from among the currently displayed placement candidates. Note that this description will mainly focus on the parts that differ from the embodiments described above, and the same or equivalent components and processes will be denoted by the same reference numerals, while redundant explanations will be omitted.

[0054] The display window 501 shown in Figure 12(a) is a settings screen displayed on the display 400 by the display control unit 206 in this embodiment. The display window 501 in Figure 12 includes an image display area 502 where an image is displayed, and a detection frame 504 is displayed for a person 503 identified as a reference subject in the image. The display window 501 in Figure 12 also displays placement candidates 505 to 510. The display control unit 206 in this embodiment identifies uncontrollable candidates from among the placement candidates 505 to 510 according to the current pan, tilt, and zoom value information transmitted from the imaging device 100 and the current position of the reference subject.

[0055] Here, we will explain how to identify uncontrollable candidates. Here, we assume that the adjustable range of the pan value is 0 degrees to +180 degrees, and that as the pan value decreases, the imaging direction changes to the left side of the horizontal image, and as the pan value increases, the imaging direction changes to the right side of the horizontal image. In this case, in the example of Figure 12(a), the current pan value of the imaging device 100 is assumed to be +10 degrees. At this time, the display control unit 206 identifies the relative position of the centroid of the person 503 with respect to the current imaging range. Then, for each of the placement candidates 505 to 510, the display control unit 206 identifies the relative position of the centroid of the icon with respect to the rectangular area of ​​the placement candidate. Then, for each of the placement candidates 505 to 510, the display control unit 206 identifies a pan value such that the relative position identified for the icon in the placement candidate and the relative position identified for the reference subject in the imaging range are approximately the same. The display control unit 206 then compares the pan value identified for each of the placement candidates 505 to 510 with the changeable range of the pan value (0 to 180 degrees), and determines that placement candidates that do not fall within the changeable range are uncontrollable candidates. For example, to achieve the composition shown by placement candidate 507, the pan value needs to be -10 degrees or +350 degrees. Since this falls outside the changeable range of the pan value (0 to +180 degrees), placement candidate 507 is determined to be an uncontrollable candidate. Similarly, in the example shown in Figure 12(a), placement candidate 510 is also determined to be an uncontrollable candidate. In the example in Figure 12, the display control unit 206 then changes the display of placement candidates 507 and 510, which were identified as uncontrollable candidates, by, for example, graying out their colors. In this way, it is possible to clearly indicate to the user that the composition of placement candidates 507 and 510 cannot be achieved. While the example of pan values ​​has been explained so far, the display control unit 206 also identifies uncontrollable candidates for tilt and zoom values ​​based on the drivable range of each value, and grays out the color of the identified position candidates, similar to the example of pan values.

[0056] In the example shown in Figure 12(a), the uncontrollable candidate is displayed in gray, but this is not the only configuration. For example, the display control unit 206 may exclude the uncontrollable candidate from the display window 501 and display only the controllable candidate, which is a controllable placement candidate other than the uncontrollable candidate, on the display window 501. In Figure 12(b), when determining the placement candidates to display in the display window 501, the display control unit 206 may identify the uncontrollable candidate from multiple placement candidates for one person as shown in Figure 6(a), according to the changeable range of the pan value, tilt value, and zoom value, as described above, and display the other placement candidates excluding the identified uncontrollable candidate on the display window 501. The placement candidates 1201 to 1202 shown in Figure 12(b) are controllable candidates newly added to Figure 12(a) after excluding the uncontrollable candidate. Now, referring to the flow shown in Figure 13, we will explain the information processing for identifying controllable placement candidates and displaying the identified multiple placement candidates on the display window 501. The processing of the flow shown in Figure 13 is executed by a functional block of the information processing device 200 shown in Figure 4, which is realized by the CPU 1400 of the imaging device 100 executing a computer program stored in the ROM 1420 of the information processing device 200.

[0057] First, the explanation for S701 to S702 shown in Figure 13 is the same as the explanation in Figure 7, so we will omit the explanation. In S1301, the display control unit 206 acquires information on the changeable range of the pan, tilt, and zoom values ​​of the imaging device 100. In S1302, the display control unit 206 identifies uncontrollable candidates and controllable candidates based on the changeable range of each value of the pan, tilt, and zoom values ​​and the position of the reference subject. If the upper limit of the number of placement candidates to be displayed is set to a predetermined value (for example, 6), the display control unit 206 identifies controllable candidates up to that predetermined value. In S1303, the display control unit 206 updates the display window 501 so that the placement candidates of the identified controllable candidates are displayed.

[0058] As described above, the information processing device 200 in this embodiment displays uncontrollable placement candidates as grayed out, or displays only controllable placement candidates. In this way, it is possible to prevent the user from selecting uncontrollable placement candidates.

[0059] (Other embodiments) Next, with reference to Figure 14, the hardware configuration of the information processing device 200 for realizing each function of each embodiment will be described. Note that while the hardware configuration of the information processing device 200 will be described in the following explanation, the imaging device 100 will also be realized with a similar hardware configuration.

[0060] The information processing device 200 in this embodiment includes a CPU 1400, RAM 1410, ROM 1420, HDD 1430, and I / F 1440.

[0061] The CPU 1400 is a central processing unit that provides overall control over the information processing unit 200. The RAM 1410 temporarily stores computer programs executed by the CPU 1400. The RAM 1410 also provides a work area used by the CPU 1400 when executing processing. Furthermore, the RAM 1410 can function, for example, as frame memory or buffer memory.

[0062] ROM1420 stores programs and other information that the CPU1400 uses to control the information processing unit 200. HDD1430 is a storage device that records image data and other data.

[0063] I / F1410 communicates with external devices via network 300, following protocols such as TCP / IP and HTTP.

[0064] In the above-described embodiments, examples are given in which the CPU 1400 performs processing. However, at least a portion of the processing performed by the CPU 1400 may be performed by dedicated hardware. For example, the processing of displaying a GUI (Graphical User Interface) or image data on the display 400 may be performed by a GPU (Graphics Processing Unit). Also, the processing of reading program code from the ROM 1420 and loading it into the RAM 1410 may be performed by a DMA (Direct Memory Access) functioning as a transfer device.

[0065] Furthermore, the present invention can also be realized by a process in which one or more processors read and execute a program that implements one or more of the functions of the above-described embodiments. The program may be supplied to a system or device having a processor via a network or storage medium. The present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more of the functions of the above-described embodiments. In addition, each part of the information processing device 200 may be realized by the hardware shown in Figure 10, or by software.

[0066] Furthermore, one or more functions of the information processing device 200 according to each embodiment described above may be provided by other devices. For example, one or more functions of the information processing device 200 according to each embodiment may be provided by the information processing device 200 itself. Furthermore, the embodiments described above may be combined, for example, by arbitrarily combining the embodiments described above.

[0067] Although the present invention has been described above along with its embodiments, these embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features. For example, combinations of the embodiments are also included in the disclosure of this specification. [Explanation of Symbols]

[0068] 100 Imaging device 200 Information Processing Devices 201 Communications Department 202 Storage section 203 Detection unit 204 Specific section 205 Operation Reception Section 206 Display Control Unit 207 Range Control Unit

Claims

1. An acquisition means for acquiring an image captured by an imaging means, A display control means that causes a plurality of arrangement candidates to be displayed on a display means, including at least a first arrangement candidate corresponding to a composition in which the subject in the imaging range of the imaging means is positioned at a first position, and a second arrangement candidate corresponding to a composition in which the subject in the imaging range is positioned at a second position different from the first position, When one of the multiple placement candidates is selected by the user, a control means performs a process to control the imaging range by controlling the imaging direction of the imaging means based on the position of the subject in the composition corresponding to the selected placement candidate and the position of the subject detected from the image. The system includes a processing means for performing grouping of multiple individuals detected from the aforementioned image, The plurality of arrangement candidates displayed on the display means are determined according to the position of the subject detected from the image or the orientation of the subject's face. The information processing apparatus is characterized in that the display control means displays on the display means one or more arrangement candidates for a first group grouped by the processing means, and one or more arrangement candidates for a second group grouped by the processing means, as the plurality of arrangement candidates.

2. The information processing apparatus according to claim 1, characterized in that the plurality of arrangement candidates to be displayed on the display control means are determined based on the position of each of the plurality of subjects detected from the aforementioned image.

3. The control means controls the imaging range by changing at least one of the pan value and tilt value of the imaging means. The information processing apparatus according to claim 1, characterized in that the display control means identifies uncontrollable candidates, which are arrangement candidates that cannot be controlled even if selected by the user, based on the position of the subject and the respective changeable ranges of the pan value and the tilt value.

4. The information processing apparatus according to claim 3, characterized in that the display control means causes the uncontrollable candidate to be displayed on the display means in a display manner different from that of other arrangement candidates.

5. The information processing apparatus according to claim 3, characterized in that the display control means displays arrangement candidates excluding the uncontrollable candidates as the plurality of arrangement candidates displayed on the display means.

6. If one of the one or more placement candidates for the first group is selected, the control means controls the imaging range according to the selected placement candidate and the positions of the subjects included in the first group. The information processing apparatus according to claim 1, wherein if one of the one or more placement candidates is selected for the second group, the control means controls the imaging range according to the selected placement candidate and the positions of the subjects included in the second group.

7. The information processing apparatus according to claim 1, characterized in that the control means can control the zoom value of the imaging means.

8. The information processing apparatus according to any one of claims 1 to 7, characterized in that the display control means causes both the image acquired by the acquisition means and the plurality of arrangement candidates to be displayed on the display means.

9. The first arrangement candidate displayed on the display means is a figure in which an icon representing a subject is placed in the region indicating the imaging range, corresponding to the first position. The information processing apparatus according to any one of claims 1 to 8, characterized in that the information of the second arrangement candidate displayed on the display means is a figure in which an icon indicating a subject is arranged in the region indicating the imaging range corresponding to the second position.

10. An acquisition step in which an image captured by an imaging means is acquired, A display control step of displaying a plurality of arrangement candidates on a display means, including at least a first arrangement candidate corresponding to a composition in which the subject is placed at a first position within the imaging range of the imaging means, and a second arrangement candidate corresponding to a composition in which the subject is placed at a second position different from the first position within the imaging range; When one of the multiple placement candidates is selected by the user, a control step is performed to control the imaging range by controlling the imaging direction of the imaging means based on the position of the subject in the composition corresponding to the selected placement candidate and the position of the subject detected from the image. The process includes a step of performing grouping of multiple people detected from the aforementioned image, The plurality of arrangement candidates displayed on the display means are determined according to the position of the subject detected from the image or the orientation of the subject's face. An information processing method characterized in that, in the display control step, one or more arrangement candidates for a first group grouped in the processing step and one or more arrangement candidates for a second group grouped in the processing step are displayed on the display means as the plurality of arrangement candidates.

11. A program for causing a computer to execute the information processing method described in claim 10.

Images