Information processing device, information processing method, and recording medium

MYPI2025003237A0Undetermined Publication Date: 2024-06-02SONY MUSIC ENTERTAINMENT (JAPAN) INC +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
MYPI2025003237
Authority / Receiving Office
MY · MY
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-12-02
Filing Date
2023-11-06
Publication Date
2024-06-02

AI Technical Summary

Technical Problem

Existing imaging control technologies face challenges in accurately tracking subjects, especially when the subject is outside the camera's view or in low-light environments, leading to difficulties in user specification and identification.

Method used

An information processing device and method that utilize a separate sensor device with a light receiving sensor to track subjects within its sensing image, allowing for subject specification and identification even when the subject is outside the camera's view or in dark environments, and a recording medium with a program to implement this functionality.

Benefits of technology

Enables accurate subject tracking and specification, improving ease of use by allowing subject identification and tracking across different environments and camera views, ensuring effective imaging control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PI2025003237_DRAWING_001
    Figure PI2025003237_DRAWING_001
  • Figure PI2025003237_DRAWING_002
    Figure PI2025003237_DRAWING_002
  • Figure PI2025003237_DRAWING_003
    Figure PI2025003237_DRAWING_003
Patent Text Reader

Abstract

An information processing device according to the present technology includes a designation reception processing unit configured to perform processing of allowing a user to designate a tracking target subject by a tracking camera capable of tracking a subject, from among subjects appearing in a sensing image obtained by a sensor device including a light receiving sensor different from the tracking camera. Figure 24
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and recording medium

[0001] The present technology relates to an information processing device and method, and a recording medium, and in particular to an imaging control technology.

[0002] Possible control related to image capture includes, for example, controlling the orientation of the camera so that it tracks a detected subject (i.e., subject tracking control), or controlling the zoom of the camera so that it matches a preset angle of view.

[0003] Patent Document 1 listed below discloses a technology for controlling the orientation of a camera so that an object is captured. Specifically, Patent Document 1 discloses a technology for detecting the position of a subject based on a captured image and also detecting the position of the subject using a wireless tag carried by the subject, and controlling the orientation of the camera based on these two detection results. Furthermore, Patent Document 2 listed below discloses a technology for controlling the imaging direction and imaging operation of a camera based on position information and audio information transmitted from an individual information transmitter attached to the subject.

[0004] JP 2008-288745 A JP 2005-277845 A

[0005] Here, when tracking a subject, it is conceivable to have the user specify in advance the subject to be tracked, but if the user specifies the subject to be tracked from among the subjects appearing in the image captured by the tracking camera that will be tracking, there is a risk that the user will not be able to properly specify the subject that they want to track. For example, depending on the situation, there is a risk that the user will not be able to properly specify the subject to be tracked, such as if the subject that the user wants to track falls outside the imaging range of the tracking camera.

[0006] The present technology has been made in consideration of the above circumstances, and aims to enable a tracking camera to appropriately specify a tracking target subject.

[0007] The information processing device according to the present technology includes a designation reception processing unit that allows a user to designate a tracking target subject from subjects captured in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera. According to the above configuration, even if the subject the user wants to track is located outside the imaging range of the tracking camera, the sensor device separate from the tracking camera can capture the subject in the sensing image, allowing the user to designate the subject they want to track from the sensing image captured by the sensor device. Furthermore, if a sensor device having a light receiving sensor capable of receiving infrared light is used, even if the user cannot identify the subject in the image captured by the tracking camera in a dark environment, the subject can be identified in the sensing image captured by the sensor device, making it easier to designate the tracking target subject.

[0008] Furthermore, an information processing method according to the present technology is an information processing method in which an information processing device performs processing to have a user designate a tracking target subject by a tracking camera capable of tracking a subject from subjects appearing in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera. Furthermore, a recording medium according to the present technology is a recording medium on which a program readable by a computer device is recorded, the recording medium recording the program causing the computer device to realize a designation acceptance processing function that performs processing to have a user designate a tracking target subject by a tracking camera capable of tracking a subject from subjects appearing in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera. These information processing methods and recording media realize the information processing device according to the present technology described above.

[0009] 1 is a diagram showing an example of the configuration of an image processing system configured including an information processing device as an embodiment according to the present technology. FIG. 1 is an image diagram of a live venue assumed in the embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of an information processing device as an embodiment. FIG. 2 is a diagram showing an example of an operation screen as an embodiment used for composition control for each camera. FIG. 3 is a diagram for explaining an example of a procedure for designating a tracking target subject. FIG. 3 is a diagram showing an example of a setting screen (in common setting mode) in the embodiment. FIG. 4 is a diagram showing an example of a setting screen (in common setting mode) on which various information is input. FIG. 4 is a diagram showing an example of a setting screen (in control setting mode) in the embodiment. FIG. 5 is a diagram showing an example of an operation screen after controller setting. FIG. 6 is a diagram showing an example of an operation screen when a tracking target subject is designated. FIG. 7 is a diagram showing an example of an operation screen when a tracking target subject is designated. FIG. 7 is a diagram showing an example of an operation screen on which a distance image obtained by a sensor device is superimposed. FIG. 8 is a diagram showing an example of a screen display for accepting designation of a tracking composition. FIG. 9 is a diagram for explaining an example of a method for calculating information for making it possible to reproduce a designated composition. FIG. 10 is an explanatory diagram of an example of a method for calculating parameter information for making it possible to reproduce a designated composition. FIG. 11 is an explanatory diagram of each case related to calculation of a target image frame. FIG. 1 is an explanatory diagram of composition types that can be switched in semi-automatic control. FIG. 1 is an explanatory diagram of a composition selection table in an embodiment. FIG. 2 is a diagram for explaining an example of composition selection based on prohibited transition composition information. FIG. 3 is a flowchart showing an example of a processing procedure for realizing composition control that selects a composition based on the composition selection table and switches to the selected composition. FIG. 4 is a flowchart of a process related to updating weight information in the composition selection table. FIG. 5 is a diagram of an example of a setting procedure for semi-automatic control. FIG. 6 is a diagram of an example of a setting procedure for semi-automatic control. FIG. 7 is an explanatory diagram of functions related to composition control as an embodiment. FIG. 8 is a flowchart showing an example of a processing procedure related to a screen display corresponding to when a tracking target subject is specified by a user. FIG. 9 is a flowchart of a process related to display switching according to the brightness of a captured image displayed on an operation screen. FIG. 10 is a flowchart of a process to be executed in response to a tracking composition specification by a user. FIG. 11 is a flowchart of a process related to calculation of a target image frame for realizing a specified tracking composition. FIG. 12 is an explanatory diagram of an overview of calibration processing as an embodiment.FIG. 1 is an explanatory diagram of a calibration processing method as a first method. FIG. 2 is a flowchart showing a specific example of the procedure for executing calibration in the first method. FIG. 3 is a flowchart of processing executed by an information processing device when the first method is adopted. FIG. 4 is an explanatory diagram of a calibration processing method as a second method. FIG. 5 is a flowchart showing a specific example of the procedure for executing calibration in the second method. FIG. 6 is an explanatory diagram of an example in which a user is prompted to specify on an image a position that will become the same point on an object. FIG. 7 is a diagram showing an example of a reception screen for setting a group shot. FIG. 8 is a diagram showing an example of an execution instruction button for instructing a tracking composition for a group shot. FIG. 9 is an explanatory diagram of a modified example in which tracking sensitivity can be variably set. FIG. 10 is a diagram showing an example of the arrangement of manual composition adjustment buttons. FIG. 11 is an explanatory diagram of an example of a composition control method in response to manual operation input. FIG. 12 is an explanatory diagram of a modified example in which a selection ratio by a switcher is displayed.

[0010] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order: <1. Image processing system as an embodiment> <2. Hardware configuration of information processing device> <3. Composition control as an embodiment> (3-1. Regarding designation of a tracking target subject) (3-2. Regarding designation of a tracking composition) (3-3. Regarding semi-automatic control) (3-4. Regarding functional configuration related to composition control as an embodiment) <4. Processing procedure> <5. Regarding calibration> <6. Modifications> <7. Recording medium> <8. Summary of embodiments> <9. The present technology>

[0011] 1 shows an example of the configuration of an image processing system 100 including an information processing device 1 according to an embodiment of the present technology. As shown in the figure, the image processing system 100 includes the information processing device 1, a parent camera 2, a tracking camera 3, a camera platform 4, a switcher 5, and a ranging / imaging device 6.

[0012] In this example, there is one parent camera 2 and multiple tracking cameras 3, and specifically, three tracking cameras 3 are used. To distinguish between the three tracking cameras 3, a hyphen (-) and a number are added to the end of the code, as shown in the figure, to represent the codes as "3-1," "3-2," and "3-3," respectively.

[0013] The parent camera 2 and the tracking camera 3 are configured as imaging devices that have imaging elements such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor and capture images.

[0014] The pan head 4 is configured as an electronic pan head, and supports the tracking camera 3, and is configured to be able to change the orientation of the tracking camera 3 in both the pan direction and tilt direction based on an external control signal. In this example, a pan head 4 is provided for each tracking camera 3, and below, when distinguishing between the three pan heads 4, they will be written as "4-1", "4-2", and "4-3" with a "- (hyphen)" and a number added to the end of the code as shown in the figure.

[0015] In the image processing system 100, a target is captured from multiple viewpoints by multiple cameras serving as the parent camera 2 and the tracking camera 3, and multiple streams of images obtained based on the captured images are input to the switcher 5. The switcher 5 selects and outputs one stream of images from the multiple streams of images input based on an operation. In this example, captured image content for a target event is generated from the images selected and output by the switcher 5. The captured image content generated based on the output images of the switcher 5 can be distributed, for example, via a network such as the Internet, or transmitted by broadcast waves. Alternatively, the captured image content can be recorded (stored) on a predetermined recording medium.

[0016] In this example, the event to be imaged by the camera is a live music event, and the parent camera 2 and tracking camera 3 are installed at the live music venue.

[0017] FIG. 2 is an image diagram of a live venue assumed in this embodiment. As shown in the figure, the live venue is equipped with a stage, an audience seating area, and an FOH (Front of House). Performers such as musicians and singers perform on the stage. The audience seating area is located behind the stage and is a space that can accommodate an audience. The FOH is located behind the audience seating area and is a space where elements related to the live performance, such as lighting, and various devices for controlling the sound of the venue are located. People from the live performance organizer, such as directors and staff, can enter the FOH.

[0018] In this example, the parent camera 2 is a camera that captures the entire stage within its field of view and is placed at the FOH. In this example, the resolution of the image captured by the parent camera 2 is 4K (3840 x 2160), while the resolution of the image output by the switcher 5 is FHD (Full High Definition: 1920 x 1080). In this example, a camera without an optical zoom is used as the parent camera 2. Two of the tracking cameras 3 are placed in the space between the stage and the audience area (the space in front of the front fence), enabling them to capture performers on stage within their field of view from a closer position than the FOH. As shown in the figure, these two tracking cameras 3 are placed at both ends in the left-right direction (the direction perpendicular to the front-to-back direction). The remaining tracking camera 3 is placed at the FOH. This tracking camera 3 is used as a camera for capturing performers on stage with a telephoto lens. In this example, each tracking camera 3 is a camera with an optical zoom. In this example, each tracking camera 3 is configured to be able to change the output resolution of the captured image. Specifically, the output resolution can be switched between at least 4K and FHD.

[0019] As will be described later, in the image processing system 100 of this example, the orientation of the tracking camera 3 is controlled so that it tracks a subject, such as a performer on stage. In other words, each tracking camera is a camera that can track a subject.

[0020] As shown in the figure, of the two tracking cameras 3 placed in front of the front fence of the stage, the tracking camera 3 placed on the downstage side (left side as you face the stage) is tracking camera 3-1, and the tracking camera 3 placed on the upstage side is tracking camera 3-2. Also, the tracking camera 3 placed at the FOH is tracking camera 3-3.

[0021] In order to enable the tracking camera 3 to track a subject, the image processing system 100 is provided with a distance measuring / imaging device 6 for detecting the three-dimensional position (position in three-dimensional space) of the subject on the stage.

[0022] The distance measuring / imaging device 6 is a type of sensor device having a light receiving sensor, separate from the tracking camera 3. Specifically, in this example, it is configured as a device having a distance measuring function and a visible light imaging function, such as a Kinect. Specifically, the distance measuring / imaging device 6 in this example has a light receiving sensor, a visible light sensor capable of receiving visible light, and an infrared light sensor capable of receiving infrared light, and has an imaging function for obtaining a visible light image based on the light receiving signal of the visible light sensor, and a distance measuring function for obtaining a distance image based on the light receiving signal of the infrared light sensor. The distance measuring / imaging device 6 in this example has a function for outputting the captured image and the distance image, and is capable of generating a bone estimation model of the subject through image recognition processing based on the distance image. The bone estimation model here refers to a simplified model of the subject that represents the subject's structure using major parts of the subject, such as the face, chest, pelvis, shoulders, elbows, hands, knees, and ankles. Hereinafter, the bone estimation model will be referred to as a "simplified model." The distance measurement and imaging device 6 also has the function of outputting information indicating the three-dimensional position of each part of the simplified model of the subject.

[0023] In this example, multiple distance measuring / imaging devices 6 are used, and are arranged at each end of the stage on the downstage and upstage. Specifically, in this example, two distance measuring / imaging devices 6 are arranged at each end of the stage on the downstage and upstage. These distance measuring / imaging devices 6 are arranged so that even if the subject moves on the stage, the subject will be captured within the distance measurement range of at least one of the distance measuring / imaging devices 6. Here, of the distance measuring / imaging devices 6, the one arranged at the front of the downstage end of the stage will be referred to as distance measuring / imaging device 6-1, and the one arranged at the front of the upstage end of the stage will be referred to as distance measuring / imaging device 6-2. Furthermore, the one arranged at the back of the downstage end of the stage will be referred to as distance measuring / imaging device 6-3, and the one arranged at the back of the upstage end of the stage will be referred to as distance measuring / imaging device 6-4.

[0024] 1, in the image processing system 100 of this example, a total of six systems of images, CAM1 to CAM6, are input to the switcher 5. For the images CAM1 to CAM3, images captured by the three tracking cameras 3 are input to the switcher 5. The images CAM4 to CAM6 are images generated by the information processing device 1 as a computer device based on images captured by the parent camera 2.

[0025] Each of the images in CAM4, CAM5, and CAM6 is an image cut out from an image captured by the parent camera 2. For these images in CAM4, CAM5, and CAM6, not only the cut-out size but also the cut-out position can be adjusted.

[0026] Here, changing the cutout position for a captured image is equivalent to virtually changing the positional relationship between the camera and the subject, and changing the cutout size for a captured image is equivalent to virtually changing the magnification of the optical zoom. In other words, obtaining an image by changing the cutout position or cutout size for a captured image can be said to be equivalent to changing the composition by moving or operating a virtual camera.

[0027] In this specification, the term "camera" is used as a concept that includes both such a virtual camera and real cameras such as the parent camera 2 and the tracking camera 3. In other words, when the term "camera" is used in this specification, it refers to a concept that includes both a real camera and a virtual camera that virtually changes the composition by cutting out a part of an image obtained by the light receiving operation of the real camera.

[0028] Furthermore, the term "imaging" is used in this specification, and this "imaging" refers to the action of obtaining an image using a camera when both a real camera and a virtual camera are defined as a "camera" as described above.

[0029] In the following description, image clipping may also be referred to as "cutout."

[0030] In this example, the image of the CAM 6 is generated by cutting out an image having a basic angle of view that is narrower than the angle of view of the image captured by the parent camera 2, for example, and output.

[0031] 1 is configured to include, for example, a computer device having a CPU (Central Processing Unit), and performs image generation for CAM4 to CAM6 as described above based on images captured by the parent camera 2, and also performs composition control for images captured by the tracking camera 3 as a real camera. Specifically, based on operation input, the information processing device 1 controls the pan and tilt of the camera platform 4 and the zoom of the tracking camera 3, thereby controlling the composition of the images captured by the tracking camera 3. Details of composition control as an embodiment will be explained again later.

[0032] 2. Hardware Configuration of Information Processing Device FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing device 1. The information processing device 1 may take the form of, for example, a personal computer. In FIG. 3, a CPU 11 of the information processing device 1 executes various processes in accordance with programs stored in a nonvolatile memory unit 14, such as a read-only memory (ROM) 12 or an electrically erasable programmable read-only memory (EEP-ROM), or programs loaded from a storage unit 19 to a random access memory (RAM) 13. The RAM 13 also stores data necessary for the CPU 11 to execute various processes, as appropriate. The CPU 11, ROM 12, RAM 13, and nonvolatile memory unit 14 are interconnected via a bus 23. An input / output interface 15 is also connected to this bus 23.

[0033] An input unit 16 consisting of operators and operation devices is connected to the input / output interface 15. For example, the input unit 16 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. The input unit 16 detects user operations, and the CPU 11 interprets signals corresponding to the input operations.

[0034] A display unit 17, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 18, such as a speaker, are connected to the input / output interface 15 either integrally or separately. The display unit 17 is a display unit that displays various types of information, and is configured, for example, by a display device provided in the housing of the information processing device 1 or a separate display device connected to the information processing device 1. The display unit 17 displays images for various types of image processing, moving images to be processed, and the like on the display screen based on instructions from the CPU 11. The display unit 17 also displays various operation menus, icons, messages, and the like, i.e., a GUI (Graphical User Interface), based on instructions from the CPU 11.

[0035] A storage unit 19 configured with a hard disk, solid-state memory, etc., and a communication unit 20 configured with a modem, etc. may be connected to the input / output interface 15. The communication unit 20 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, etc.

[0036] A drive 21 is also connected to the input / output interface 15 as needed, and a removable recording medium 22 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately attached. The drive 21 allows data files such as image files and various computer programs to be read from the removable recording medium 22. The read data files are stored in the storage unit 19, and images and sounds contained in the data files are output on the display unit 17 and the audio output unit 18. In addition, computer programs and the like read from the removable recording medium 22 are installed in the storage unit 19 as needed.

[0037] In this information processing device 1, software can be installed via network communication by the communication unit 20 or via a removable recording medium 22. Alternatively, the software may be stored in advance in the ROM 12, the storage unit 19, or the like.

[0038] 4 is a diagram showing an example of an operation screen Gm as an embodiment used for composition control for each camera. In this example, the operation screen Gm is a screen displayed on the display unit 17 of the information processing device 1, and as shown in the figure, an execution instruction button Bm for issuing an instruction to execute composition control is arranged for each camera CAM1 to CAM6. Specifically, the operation screen Gm of this example is provided with a camera name display area Arc that displays name information for each camera CAM1 to CAM6, and an execution instruction button Bm is arranged for each camera. A user such as a director can operate the execution instruction button Bm displayed for a certain camera to instruct the information processing device 1 to execute composition control for that camera in a manner determined corresponding to the operated execution instruction button Bm.

[0039] In this example, two items related to composition control are defined: "Production" and "Target." The operation screen Gm is provided with a production area Are for placing the execution instruction button Bm related to "Production" for each camera, and a target area for placing the execution instruction button Bm related to "Target" for each camera.

[0040] The composition control for the "performance" is a composition control based on imaging performance information. The imaging performance information can be said to be information indicating changes in a captured image, and more specifically, information indicating the visual performance aspect of the captured image. Examples of composition control for this "performance" include control of a "gradual zoom" (a slow zooming technique often used to capture live music performances) and a "handheld feel" (the effect of capturing images with a handheld camera). Four types of gradual zoom performances are available: "pull-to-close" (i.e., zooming in), "close-to-pull" (zooming out), "alternate," and "random" (randomly switching between pull-to-close and close-to-pull).

[0041] The user can specify in advance which type of performance execution instruction button Bm to place for each camera in the performance area Are using the "control settings" on the setting screen Gs (described later) (see FIG. 8).

[0042] "Target" is an item related to specifying a subject to be tracked. By operating an execution instruction button Bm belonging to this "Target" item, composition control is performed to realize a composition that includes the subject corresponding to the execution instruction button Bm in the captured image.

[0043] In this example, the composition control of the "target" includes control that uses the position detection results of the subject and control that does not. Composition control that uses the position detection results of the subject can be, for example, subject tracking control. In this case, the execution instruction button Bm is associated with the subject to be tracked, and by operating the execution instruction button Bm, control is performed to realize a composition that tracks the associated subject. For example, if the performers to be imaged are members of a rock band, the "target" execution instruction button Bm can be associated with the performer as a "vocalist" or a "guitarist," and the user can operate the execution instruction button Bm to issue an instruction to execute control that realizes a composition that tracks the performer as a "vocalist" or a "guitarist."

[0044] Furthermore, an example of composition control that does not use the subject position detection results is control according to a "fixed position," which will be described later. As will be described later, the "fixed position" is a function that allows a specific position within the imaging target space to be preset as the position of the imaging target. By operating the "fixed position" execution instruction button Bm, the user can issue an instruction to execute control that realizes a composition in which the preset specific position fits within the captured image (e.g., a composition in which the specific position coincides with a predetermined position such as the center of the captured image). For example, in the figure, the execution instruction buttons Bm for "drum left," "drum right," "drum center," and "pull back" correspond to the "fixed position" execution instruction button Bm. As can be understood from the above explanation, the imaging target (target) in the composition control of this embodiment includes not only a specific person but also a specific position.

[0045] In this embodiment, the user can arbitrarily specify the tracking composition of the camera for tracking a subject. Specifically, the user can arbitrarily specify the position and size (zoom amount) of the tracking target within the image frame. Therefore, when an execution instruction button Bm, which is arranged in the target area Art in the figure and instructs control to realize a composition for tracking a target subject such as "vocals," "guitar," or "bass," is operated, pan, tilt, and zoom control of the target camera is performed so that a tracking composition previously specified for the target subject is realized.

[0046] Here, on the operation screen Gm, the information processing device 1 (CPU 11) performs processing to display the execution instruction button Bm for which composition control is being executed and the execution instruction button Bm for which composition control is not being executed in different display modes, for both the execution instruction button Bm located in the performance area Are and the execution instruction button Bm located in the target area Art, such as highlighting the execution instruction button Bm for which operation has been performed and the corresponding composition control is being executed.

[0047] Furthermore, in this example, a tracking indicator It is provided for an execution instruction button Bm in the target area Art of the operation screen Gm, which is used to issue an instruction to execute composition control for tracking a target subject. For an execution instruction button Bm provided with a tracking indicator It, the information processing device 1 performs processing to change the display mode of the tracking indicator It depending on whether the distance measurement / image capture device 6 has detected the tracking target subject corresponding to the execution instruction button Bm. For example, if any of the distance measurement / image capture devices 6 has detected the tracking target subject, the display color of the tracking indicator It is a specific color such as blue, and if the tracking target subject has not been detected by any of the distance measurement / image capture devices 6, the display color of the tracking indicator It is a color other than the specific color, such as gray.

[0048] Furthermore, for the execution instruction button Bm arranged in the effect area Are, a display indicating the level of composition control related to the effect executed by operating the execution instruction button Bm can be displayed (see the numbers in the squares in the figure). For example, for the gentle zoom, the zoom distance can be set in, for example, 10 stages on the setting screen Gs described below, so it is conceivable to display a numerical value indicating the stage of the zoom distance. Furthermore, for the hand feeling, the level can be set in, for example, 10 stages on the setting screen Gs, so it is conceivable to display a numerical value indicating the level.

[0049] The operation screen Gm is provided, for example, at the top of the screen, with a controller name display area Acn, a pull-down button Bpd, a tracking target display area At, and a menu button B0. The controller name display area Acn is an area that displays the name information of the controller. It is desirable that which execution instruction button Bm is arranged for each camera can be switched for each imaging target, such as for each artist being imaged. In this example, in the setting screen Gs described below, it is possible to set multiple combinations of the arrangement of the execution instruction buttons Bm for each camera as "controllers." When multiple controllers are set, the user can specify the controller to be called up on the operation screen Gm from a list of controllers by operating the pull-down button Bpd on the operation screen Gm.

[0050] The tracking target display area At is an area that displays identification information for each of the number of subjects that can be designated as tracking targets. In this example, the number of subjects that can be designated as tracking targets is set to "6." Accordingly, the tracking target display area At includes six individual display areas, t1 to t6, that display identification information for each tracking target subject. As shown in the figure, each individual display area includes an image display area ti and a name display area tn. As will be described later, in this embodiment, the tracking target subjects are designated from among the subjects appearing in the image captured by the ranging / imaging device 6. FIG. 4 illustrates a state after up to five of the six subjects that can be designated as tracking targets have been designated as tracking targets. In the image display area ti for a subject for which a tracking target has been designated, an image captured by the ranging / imaging device 6 for the designated subject is displayed. Here, the composition of the image of the corresponding tracking target subject displayed in the image display area ti is a composition that captures the upper half of the subject. Furthermore, if the distance measurement / image capture device 6 loses the tracking target subject, the image display area ti will be switched to display a specific image other than the image captured by the distance measurement / image capture device 6 (for example, an icon image imitating the upper half of a person, displayed in the image display area ti of the sixth tracking target subject in the figure; hereinafter referred to as a "lost notification image").

[0051] In each individual display area in the tracking target display area At, the name display area tn displays name information set in a setting screen Gs, which will be described later. Note that if name information has not been set, a predetermined initial character string will be displayed, such as "No Name" in the figure.

[0052] Also, on the operation screen Gm, a semi-auto ON / OFF button B1 is a button for instructing ON / OFF of semi-auto composition control, which will be described later.

[0053] The menu button B0 is a button for calling various menus.

[0054] Here, on the operation screen Gm, in the camera name display area Arc, it is also possible to display the camera being selected by the switcher 5 (the camera currently outputting PGM) and the so-called NEXT (Preview) camera (the camera that will be outputting PGM next to the camera currently outputting PGM: the candidate camera for PGM output) (see the hatched and pear-textured areas in the figure).

[0055] 3. Composition Control as an Embodiment (3-1. Designation of Tracking Target Subject) The procedure for designating a tracking target subject will be described with reference to Fig. 5 to Fig. 12. To start designating a tracking target subject, as shown in Fig. 5, the menu button B0 on ​​the operation screen Gm is operated, and an operation for selecting the "Settings" item from the called menu M is performed.

[0056] 6 shows an example of a setting screen Gs that is displayed on the display unit 17 in response to a selection operation of the "Settings" item. As shown in the figure, the setting screen Gs is provided with a common setting button B2, a control setting button B3, and a start button Bs, as well as a camera switching operation area As1, a subject name setting area As2, and a controller setting area As3. FIG. 6 shows an example of the setting screen Gs that is displayed when the common setting button B2 is selected out of the common setting button B2 and the control setting button B3.

[0057] The camera switching operation area As1 is provided with a camera switching button Bc and a camera name input box bcm. The camera switching button Bc allows the camera for which the camera name is to be set to be switched between CAM1 and CAM6. In the camera switching operation area As1, after selecting the camera for which you want to set a name with the camera switching button Bc, you can enter name information in the camera name input box bcm to set name information for the selected camera.

[0058] The subject name setting area As is provided with subject name input boxes (bt1 to bt6) equal to the settable number of tracking target subjects. As described above, in this example, the settable number of tracking target subjects is six, and subject name input boxes bt1 to bt6 are provided. Name information input into these subject name input boxes is reflected in the corresponding name display area tn in the tracking target display area At of the operation screen Gm.

[0059] The controller setting area As3 displays controller slots, and provides a controller name input box bcn and a controller delete button Be1 within the slots. The controller name input box bcn allows the name information of the controller corresponding to that slot to be set. Furthermore, operating the delete button Be1 can instruct the deletion of the controller corresponding to that slot. Furthermore, the controller setting area As3 provides an add controller button Bp1, and operating the add button Bp1 can instruct the addition of a new controller slot.

[0060] FIG. 7 shows an example of the state of the setting screen Gs after the camera name, the name of the subject to be tracked, and the controller name have been set from the state shown in FIG.

[0061] When the start button Bs is operated on the setting screen Gs, the screen transitions to an operation screen Gm displaying the combination of execution instruction buttons Bm for each camera set for the controller. At this time, the operation screen Gm displays information about the controller name set for the selected controller in the controller name display area Acn.

[0062] Here, on the setting screen Gs, by operating the control setting button B3 to transition to the control setting mode, it is possible to set the arrangement and combination of the execution instruction buttons Bm for each camera, that is, to set the controller.

[0063] Fig. 8 shows an example of a setting screen Gs in the control setting mode. The setting screen Gs in the control setting mode is provided with a common setting button B2, a control setting button B3, and a start button Bs, similar to the setting screen Gs in the common setting mode shown in Fig. 6, but is also provided with a camera switching button Bc, an operation area controller name display area Acn, a pull-down button Bpd, a performance setting area As4, and a preset setting area As5.

[0064] The effect setting area As4 is provided with check boxes cb that allow individual specification of composition control related to the effect, specifically, "pull back to close in," "random," "close back to pull back," "alternate," and "handheld feel" related to the "gradual zoom" effect in this example. The effect setting area As4 also has a setting operation unit, such as a slide bar, for setting the zoom distance (zoom amount) for at least the "gradual zoom." The strength of the "handheld feel" is expressed as a change in at least one of the magnitude of blur in the composition and the speed of the blur.

[0065] The preset setting area As5 displays preset slots. Here, setting a preset corresponds to setting an execution instruction button Bm to be placed in the target area Art on the operation screen Gm. Specifically, it is setting composition control content to be associated with the execution instruction button Bm.

[0066] The preset slot is provided with a preset name input box bpn for setting a preset name (i.e., name information to be displayed on the execution instruction button Bm), a check box cbs for specifying composition control for capturing the aforementioned "fixed point position" as a composition control mode, and a check box cbt for specifying composition control for tracking a subject. When the latter check box cbt is checked, a check box cb for selecting which subject to track is displayed in the preset name slot. The check boxes cb displayed here are check boxes cb for the tracking target subjects whose name information has been set in the subject name setting area As2 (see FIG. 6 ) described above. In the example of FIG. 7 , name information has been set for five people: vocalist, guitarist, bassist, drummer, and keyboardist, and therefore, the example shown here shows check boxes cb (cb1 to cb5) for these five people.

[0067] In the preset setting area As5, a delete button Be2 is provided in each preset slot for deleting the preset corresponding to that slot. Also provided in the preset setting area As5 is an add button Bp2 for instructing the addition of a preset slot.

[0068] In the setting screen Gs during this control setting mode, the user can switch the camera to be set using the camera switching button Bc, and perform operations on the performance setting area As4 and preset setting area As5, thereby setting the execution instruction button Bm to be displayed on the operation screen Gm for the desired camera.

[0069] Here, an example is given in which the execution instruction button Bm to be displayed on the operation screen Gm is set for a controller set with the name "Artist ○△". However, if multiple controller slots have been generated in the controller setting area As3 (see Figure 6) described above, the user can call up another controller on the setting screen Gs by operating the pull-down button Bpd on the setting screen Gs shown in Figure 8 and set the execution instruction button Bm in the same way.

[0070] 9 shows an example of the operation screen Gm after the controller is set up. Specifically, the operation screen Gn is displayed in response to the start button Bs being operated after the controller of the "artist ○△" is set up on the setting screen Gs shown in FIG.

[0071] In this state, the name information of the tracking target subject is set, but the specific subject to be tracked has not been set. In this embodiment, the specification of the specific subject to be tracked, that is, the specification of the tracking target subject, is performed in the following manner. That is, the tracking target subject is specified by the user from the subjects appearing in the sensing image obtained by the distance measurement / image capture device 6 (that is, a sensor device having a light receiving sensor separate from the tracking camera 3).

[0072] In this case, the user first performs an operation (e.g., a tap operation) to select an individual display area of ​​a subject to be designated in the tracking target display area At of the operation screen Gm. Here, it is assumed that an operation to select the individual display area t1 of the subject designated as a “vocalist” has been performed (see <1> in the figure).

[0073] When an operation to select any of the individual display areas in the tracking target display area At of the operation screen Gm is performed, the information processing device 1 performs a process of superimposing and displaying the captured image Ims by the distance measuring / imaging device 6 on the operation screen Gm, as shown in Fig. 10. Specifically, in this example, the captured image Ims is superimposed and displayed on the performance area Are. Here, there are four distance measuring / imaging devices 6, and while this is an example, only the captured images Ims from three of the devices are displayed, it is of course possible to display the captured images Ims from all four devices.

[0074] At this time, the information processing device 1 performs subject recognition processing on the captured images Ims input from each distance measurement / image capture device 6, and displays a detection frame Fd for the subject appearing in each captured image Ims in a superimposed manner.

[0075] The user performs an operation to designate the subject (vocalist in this example) selected in the previous operation in FIG. 9 from the subjects appearing in the captured images Ims displayed on the operation screen Gm in this way (see <2> in the figure). Specifically, this designation operation is an operation to select the detection frame Fd of the corresponding subject, specifically, for example, an operation to tap inside the detection frame Fd. This designation operation of the detection frame Fd only needs to be performed for at least one captured image Ims.

[0076] By performing such a selection operation of the detection frame Fd, it is possible to specify which of the subjects in the individual display area selected in the previous operation <1> should be the tracking target.

[0077] In response to the selection operation of the detection frame Fd, the information processing device 1 performs a process of displaying name information of the subject designated by the selection operation of the detection frame Fd, i.e., the subject designated as the tracking target, at a position corresponding to the detection frame Fd of the subject, as shown in Fig. 11. Furthermore, in response to the selection operation of the detection frame Fd, the information processing device 1 performs a process of displaying an image of the tracking target subject extracted from the captured image Ims in the image display area ti of the individual display area selected by the operation <1> above, as exemplified in Fig. 11 (see <3> in the figure). As will be understood from the above description, the image display in the image display area ti displays an image in which the upper half of the tracking target subject in the captured image Ims is extracted.

[0078] Here, the name information of the tracking target subject in the detection frame Fd can be displayed not only for the image in which the detection frame Fd is selected, but also for all captured images Ims in which the tracking target subject appears.

[0079] By displaying name information in the detection frame Fd and displaying an image in the image display area ti as described above, the user can intuitively understand whether or not the tracking target subject has been designated as intended.

[0080] Here, the detection frames Fd for subjects designated as tracking target subjects and the detection frames Fd for subjects not designated as tracking target subjects can be displayed in different display modes. For example, designated can be displayed in a specific color such as blue, and undesignated can be displayed in a non-specific color such as gray. Alternatively, designated can be displayed in a solid frame and undesignated can be displayed in a dotted frame, or designated can be displayed in a thick frame and undesignated can be displayed in a thin frame.

[0081] The user can similarly specify other subjects for which name information has been set as the tracking target subject from the captured image Ims by selecting the individual display area of ​​the tracking target display area At and selecting the detection frame Fd in the captured image Ims.

[0082] In this embodiment, the information processing device 1 determines whether the brightness of the captured image Ims displayed on the operation screen Gm for designating the tracking target subject is equal to or less than a predetermined brightness, and, if the brightness becomes equal to or less than the predetermined brightness, performs processing to display, as an image for accepting designation of the tracking target subject, an image based on the distance image captured by the distance measurement / image capture device 6. Specifically, in this example, if the brightness of the captured image Ims becomes equal to or less than the predetermined brightness, the information processing device 1 performs processing to display the distance image captured by the distance measurement / image capture device 6 by superimposing it on the captured image Ims, as exemplified in FIG.

[0083] The brightness of the captured image Ims may be determined based on, for example, the average brightness value within the image. Also, instead of the distance image, a simple model generated based on the distance image may be displayed.

[0084] Displaying an image based on the distance image as described above makes it easier for the user to specify a tracking target subject even in a dark environment. In particular, it is expected that the operation of specifying a tracking target subject will be performed when the artist to be imaged comes on stage but before they start performing, and since this is often a dark environment with the lights on the stage turned off, it is effective to display an image based on the distance image as described above.

[0085] (3-2. Designation of Tracking Composition) Next, designation of a tracking composition will be described with reference to Figs. 13 to 16. In this embodiment, it is possible to designate a tracking composition for each tracking target subject. When designating a tracking composition for a certain tracking target subject, the user performs an operation to select the subject for which the tracking composition is to be designated in the subject name setting area As2 on the setting screen Gs in the common setting mode shown in Fig. 7.

[0086] In response to this selection operation, the information processing device 1 performs processing to display a composition designation screen as shown in Fig. 13A on the display unit 17. This composition designation screen displays a virtual simple model Mbv of the subject, an image frame Fs, a reset button B5, and an end button B6. Here, the virtual simple model Mbv is a virtual simple model of the subject.

[0087] In this embodiment, the information processing device 1 accepts the designation of the tracking composition as the designation of a tracking image frame for the display image of the virtual simple model Mbv. Specifically, in this example, the image frame Fs side is fixed, and an adjustment operation of the size and position of the virtual simple model Mbv within the image frame Fs is accepted as an adjustment operation of the tracking composition.

[0088] In this example, the composition adjustment is performed by operating the keys on the keyboard. Specifically, the operation of the + key and the - key is an operation for enlarging or reducing the virtual simple model Mbv within the image frame Fs, and the operation of the up, down, left, and right arrow keys is an operation for adjusting the position of the virtual simple model Mbv within the image frame Fs.

[0089] In Figure 13, the transition from Figure 13A to Figure 13B shows that an enlargement operation has been performed using the + key, and the transition from Figure 13B to Figure 13C shows that the virtual simple model Mbv has been shifted to the right within the image frame Fs using the right arrow key.

[0090] If the user wishes to specify the adjusted composition, the user operates the end button B6, or the reset button B5 to reset the composition to the initial state shown in Fig. 13A.

[0091] For confirmation, it is possible to specify the tracking composition before specifying the tracking target subject from the captured image Ims as described with reference to FIGS.

[0092] Furthermore, on the composition specification screen for the tracking composition, if zooming based on the center point is not possible when zooming is changed at a boundary value, it is possible to zoom by moving the center point of the zoom to a range that includes the virtual simple model Mbv.

[0093] When the above-mentioned end button B6 is operated to designate a tracking composition, the information processing device 1 performs a process of calculating and storing parameter information for making it possible to reproduce the designated composition.

[0094] Fig. 14A schematically shows a specified tracking composition. When a tracking composition is specified, the information processing device 1 determines an intersection J as shown in Fig. 14B. This intersection J is the intersection of a vertical center line Vc (a line that vertically bisects the image frame Fs), which is the center line of the image frame Fs in the vertical direction, and a horizontal center line (a line that horizontally bisects the virtual simple model Mbv) of the virtual simple model Mbv. Note that Fig. 14B shows the center Fc of the image frame Fs and the horizontal center line Hc (a line that horizontally bisects the image frame Fs).

[0095] As parameter information for reproducing the specified composition, the information processing device 1 calculates the vertical positional deviation Vrc shown in Fig. 15A and the horizontal offset Hof shown in Fig. 15B. The vertical positional deviation Vrc is a value obtained by scaling the vertical position of the intersection point J using a vertical scaler (hereinafter referred to as the "first vertical scaler") that is defined such that the vertical position of the head of the virtual simple model Mbv is 1.0 and the vertical position of the ankle is 0.0. In other words, the vertical positional deviation Vrc is a value that represents the vertical position of the intersection point J as a relative value, with the length from the ankle to the head of the virtual simple model Mbv being 1.0.

[0096] In this example, the tracking composition can be specified within a range that satisfies the condition that at least a portion of the virtual simple model Mbv fits within the image frame Fs, so the vertical positional deviation amount Vrc may be a value exceeding 1.0 or a value smaller than 0.0 (i.e., a negative value).

[0097] The horizontal offset amount Hof is calculated as a value obtained by scaling the horizontal position of the intersection point J using a horizontal scaler in which the left end position of the image frame Fs is 0.0 and the right end position is 1.0, as shown in FIG. 15B.

[0098] The information processing device 1 also stores information on the zoom amount Zr as parameter information for reproducing the specified composition. The information on the zoom amount Zr is stored as the vertical length of the image frame Fs in the first vertical scaler, which is specified by the zoom operation of the virtual simple model Mbv performed on the composition specification screen, expressed as a value between the minimum allowable zoom amount of 0.0 and the maximum allowable zoom amount of 1.0.

[0099] The maximum allowable zoom amount may be considered to be the amount at which the vertical length of the image frame Fs is twice the vertical length of the virtual simple model Mbv, and the minimum allowable zoom amount may be considered to be the amount at which the vertical length of the image frame Fs is 1 / 5 times the vertical length of the virtual simple model Mbv.

[0100] Next, a method for calculating a target image frame for reproducing a specified tracking composition will be described. In this example, the target image frame is calculated as a target image frame in a world coordinate system (three-dimensional spatial coordinate system) based on the distance measurement / image capture device 6, rather than in the camera coordinate system of the tracking camera 3. Specifically, the vertical and horizontal center coordinates of the target image frame are calculated.

[0101] The calculation method of the target image frame differs depending on the following cases 1 to 4. FIG. 16 is an explanatory diagram of cases 1 to 4. Case 1 is a case where the value of the vertical positional deviation Vrc is greater than the value (1.0) of the head position of the virtual simple model Mbv on the first vertical scaler. Case 2 is a case where the value of the vertical positional deviation Vrc is equal to or less than the value of the head position of the virtual simple model Mbv on the first vertical scaler and greater than the value of the pelvis position. Case 3 is a case where the value of the vertical positional deviation Vrc is equal to or less than the value of the pelvis position of the virtual simple model Mbv on the first vertical scaler and greater than the value of the ankle position. Case 4 is a case where the value of the vertical positional deviation Vrc is smaller than the value of the ankle of the virtual simple model Mbv on the first vertical scaler.

[0102] In all of Cases 1 to 4, the vertical center coordinate of the target image frame is calculated using a similar method based on the three-dimensional position information of the detection simple model Mbd, which is a simple model of the tracking target subject generated based on the distance image captured by the distance measurement / image capture device 6, and the vertical positional deviation Vrc. Specifically, to calculate the vertical center coordinate of the target image frame, a vertical scaler (hereinafter referred to as the "second vertical scaler") is used that sets the vertical position of the head position of the detection simple model Mbd to 1.0 and the vertical position of the ankle position to 0.0. In this case, the vertical position of the ankle used as the reference in the second vertical scaler is set to the vertical position of the lower of the left and right ankles, taking into account the possibility that the tracking target subject may have one foot raised.

[0103] The vertical center coordinate of the target image frame is determined by the vertical position specified by the vertical position shift amount Vrc in the second vertical scaler. That is, if the value of the vertical position shift amount Vrc is 0.8, for example, the vertical coordinate of the position where the value is 0.8 in the second vertical scaler is determined as the vertical center coordinate of the target image frame.

[0104] Furthermore, the horizontal center coordinate of the target image frame is determined as follows for each of Cases 1 to 4. That is, in Case 1, the horizontal center coordinate of the target image frame is determined so that the amount of horizontal offset from the horizontal coordinate of the head position in the simple detection model Mbd of the tracking target subject matches the horizontal offset amount Hof.

[0105] In case 2, the horizontal center coordinates of the target image frame are determined so that the amount of horizontal offset from the horizontal center line of the detected simple model Mbd of the tracking target subject matches the horizontal offset amount Hof.

[0106] In case 3, the horizontal center coordinate of the target image frame is determined so that the horizontal offset amount from the horizontal coordinate of the pelvis center position in the detected simple model Mbd of the tracking target subject matches the horizontal offset amount Hof.

[0107] In case 4, as in case 3 above, the horizontal center coordinate of the target image frame is determined so that the horizontal offset amount from the horizontal coordinate of the pelvic center position in the detected simple model Mbd of the subject to be tracked matches the horizontal offset amount Hof.

[0108] Here, in any of Cases 1 to 4, the size of the target image frame is set to a size specified by the zoom amount Zr in the second vertical scaler.

[0109] By employing the composition reproduction method based on relative values ​​using the virtual simple model Mbv and the detected simple model Mbd as described above, it is possible to ensure that the designated tracking composition is properly reproduced even when the size of the virtual simple model Mbv and the size of the actual detected simple model Mbd (the size of the tracking target subject) differ. In other words, it is possible to ensure that the designated tracking composition is properly reproduced regardless of the size of the tracking target subject.

[0110] Here, as described above, the center coordinates of the target image frame are set as coordinate information in the world coordinate system. To enable conversion of this into coordinate information in the camera coordinate system of the tracking camera 3 to be controlled, the positional relationship between at least each distance measurement / image capture device 6 and each tracking camera 3 is grasped by a calibration process described later. By grasping the positional relationship between each distance measurement / image capture device 6 and each tracking camera 3, it is possible to obtain a coordinate conversion table for converting coordinates in three-dimensional space into coordinates in the camera coordinate system of each tracking camera 3. Once the coordinates in the camera coordinate system for the center position of the target image frame are obtained, it is possible to calculate pan and tilt angle information for aligning the optical axis center with those coordinates, and it is possible to control the tracking camera 3 so as to realize a specified tracking composition.

[0111] In the above example, the first vertical scaler is a scaler that sets the vertical position of the head of the virtual simple model Mbv to 1.0 and the vertical position of the ankle to 0.0, but the first vertical scaler may be a scaler that sets the vertical position of a predetermined part of the virtual simple model Mbv as a reference value and that uses the length between the predetermined parts separated in the vertical direction on the virtual simple model Mbv as a reference unit.Furthermore, the second vertical scaler may be a scaler that sets the vertical position of the predetermined part of the detected simple model Mbd as a reference value and that uses the length between the predetermined parts on the detected simple model Mbd as a reference unit.

[0112] Just to be clear, even when an execution instruction button Bm is selected to instruct tracking of a tracking target subject for which a tracking composition has been specified, and tracking of the tracking target subject is being performed, if an execution instruction button Bm is selected to instruct the execution of an effect such as a gradual zoom or a handheld feel, composition control related to the corresponding effect will also be executed.

[0113] (3-3. Regarding semi-automatic control) The information processing device 1 of this embodiment is capable of performing semi-automatic composition control, which automatically switches the subject to be tracked and the tracking composition of each camera, without the user having to select the execution instruction button Bm on the operation screen Gm.

[0114] 17A to 17D are explanatory diagrams illustrating composition types that can be switched in semi-automatic control. Composition types that can be switched in semi-automatic control include "UP (close-up shot)," "BS (bust shot)," "WS (waist shot)," and "FF (full figure)," as shown in FIGS. 17A to 17D. "UP" is a composition in which the face of a person as the subject (image subject) is fully captured within the image frame, while "BS" is a composition in which only the portion of the person as the subject from the chest to the top of the head is captured within the image frame. Furthermore, "WS" is a composition in which only the portion of the person as the subject from the waist to the top of the head is captured within the image frame, while "FF" is a composition in which the entire person as the subject from the head to the feet is captured within the image frame.

[0115] In the semi-automatic composition control, the "composition" is specified by the "composition type" and the type of subject to be imaged (the performer in this example) as shown in Fig. 17. For example, the composition is specified as a composition that captures the subject as a guitar player with the composition type = "WS", or a composition that captures the subject as a vocalist with the composition type = "UP".

[0116] In semi-automatic control, the camera selects a composition based on a composition selection table.

[0117] 18 is an explanatory diagram of a composition selection table. The composition selection table is information in which weight information is associated with each combination of a subject to be captured by a camera and a composition type of the camera. Here, an example of a composition selection table is shown for a case in which the event to be captured is a live music event, and the selectable subjects to be captured are the vocalist, guitarist, or bassist of a music band. The weight information associated with each combination of a subject to be captured and a composition type, in other words, the weight information associated with each composition specified by the combination of a subject to be captured and a composition type, can be said to be information indicating the selection rate of that composition.

[0118] In this embodiment, a composition selection table is prepared for each camera, and the information processing device 1 selects a composition using the corresponding composition selection table for each camera 3. In this example, the composition selection table is stored in a predetermined memory device such as the storage unit 19, and the information processing device 1 (CPU 11) selects a composition for each camera based on the composition selection table thus stored.

[0119] It is conceivable to prepare multiple types of composition selection tables, each with a different combination of composition type and weight information, for each camera targeted for composition control. For example, a composition selection table may be prepared for each of music producers A, B, and C, each with a different combination of composition type and weight information so that the composition frequently used by each music producer can be easily selected. The user may then be able to select the table to use for composition selection from among these composition selection tables. This allows the user to select which music producer's style the captured image content will be finished as.

[0120] In the semi-automatic control, the composition selection for each camera is performed based on the prohibited transition composition information so as to prevent the occurrence of a predetermined prohibited transition of the composition. The prohibited transition here refers to a prohibited transition of the composition between images sequentially selected by the switcher 5.

[0121] Based on the prohibited transition composition information and selection history information (corresponding to composition history information) of the camera by the switcher 5, the information processing device 1 selects a composition based on the composition selection table so that the composition transition between images sequentially selected by the switcher 5 does not become a prohibited transition defined in the prohibited transition composition information. Specifically, based on the prohibited transition composition information and the composition of the camera currently selected by the switcher 5, the information processing device 1 identifies, from among the compositions in the composition selection table of each camera, a composition that will become a prohibited transition if next selected by the switcher 5, as a "prohibited transition composition," and performs composition selection based on the weight information of the composition selection table, targeting only compositions excluding this prohibited transition composition.

[0122] An example of composition selection based on such prohibited transition composition information will be described with reference to FIG. 19 . Specifically, FIG. 19 illustrates an example of composition selection for a certain camera in a state where the composition of the camera currently selected by the switcher 5 is “bass-WS,” and transitions from the “bass-WS” composition to the “bass-UP” composition and transitions to a composition with the “bass” as the imaging subject are defined as prohibited transitions. In this case, the composition selection table defines the “bass-UP” composition and the composition with the “bass” as the imaging subject as prohibited transition compositions, so the information processing device 1 selects compositions excluding these compositions (in the illustrated example, a composition with the vocalist as the imaging subject, a “bass-WS” composition, and a “bass-FF” composition). This prevents the composition transition of the output image by the switcher 5 from becoming a prohibited transition, even if the target camera is switched to the selected composition and the image captured by that camera is selected by the switcher 5.

[0123] Furthermore, in semi-automatic control, the information processing device 1 executes composition selection based on the composition selection table for each camera described above, on the condition that image selection has been performed by the switcher 5. In other words, when a new image from one of the captured images from each camera is selected by the switcher 5, composition selection is performed for each camera based on the composition selection table accordingly, and the composition of each camera is controlled to the selected composition.

[0124] Furthermore, in semi-automatic control, the information processing device 1 selects a composition based on the composition selection table, excluding cameras for which images are being selected by the switcher 5. As described above, in this example, composition selection is performed based on the composition selection table on the condition that image selection has been performed by the switcher 5, but in this case, composition selection is performed based on the composition selection table, excluding cameras for which images are being selected by the switcher 5. By excluding cameras for which images are being selected by the switcher 5 from the targets for composition selection as described above, it is possible to prevent composition switching from being performed for a camera selected by the switcher 5 even when a captured image is being selected.

[0125] Furthermore, under semi-automatic control, the information processing device 1 selects a composition based on the composition selection table for only the musical accompaniment section identified from the audio analysis results of the event to be captured. As an example, each camera is equipped with a microphone, and audio data collected by the microphone is attached to the captured image data of each camera. The information processing device 1 then performs audio analysis on the attached audio data to determine whether or not the section corresponds to a musical accompaniment section. Based on the determination result, the information processing device 1 selects a composition based on the composition selection table for only the musical accompaniment section. Note that the method of acquiring audio data is not limited to the method using the camera's microphone as described above. For example, audio data collected by a microphone provided in a device other than the camera, such as a PA (Public Address) device, may also be used for audio analysis. In this case, the audio data is not attached to the captured image of the camera, but is treated as data independent of the captured image. Note that the determination of whether or not the section corresponds to an audio accompaniment section may also be performed based on a user's operational input.

[0126] In the case of captured image content from a music event, for example, between songs, such as during a Master of Ceremonies (MC) segment, there is less need for composition changes compared to during song segments. By selecting compositions only for the musical accompaniment segments as described above, unnecessary composition changes can be prevented even between songs, and the processing load associated with composition changes can be reduced.

[0127] Furthermore, in semi-automatic control, the information processing device 1 updates the weight information in the composition selection table based on preset conditions. As an example, the information processing device 1 updates the weight information based on selection history information of the camera by the switcher 5. In this case, each time the switcher 5 selects a camera, in other words, each time a captured image is selected, the information processing device 1 stores information about the captured image and its composition as selection history information in a predetermined storage device such as the storage unit 19, and updates the weight information in the composition selection table based on the selection history information.

[0128] A specific method of updating weights based on selection history information is, for example, to reduce the weight of the weight information for compositions that are frequently selected by the switcher 5 (selection frequency equal to or greater than a certain frequency) among the weight information for each composition in each composition selection table. In this way, the weights are updated so that the switcher 5 is less likely to select the same composition.

[0129] The weight information may also be updated based on the content of the event to be captured. Specifically, if a solo part of an instrument, such as a guitar solo, is detected as the content of the event to be captured, the weight information may be updated to increase the weight of the composition in which the player of that instrument is the capture target. In this example, the detection of the solo part of the instrument is performed, for example, based on the results of an audio analysis of the event to be captured. Specifically, as described above, the detection is performed based on the results of an audio analysis of audio data accompanying the images captured by each camera or audio data obtained by PA equipment, etc. Note that the detection of the solo part of the instrument can also be performed based on a user's operational input.

[0130] By increasing the weight of the composition that targets the player of an instrument in the image in response to the detection of the solo part of that instrument as described above, it becomes easier to select an appropriate composition that matches the content of the event, thereby improving the quality of the captured image content.

[0131] Here, there are at least two possible examples of when to update the weight information. In the first example, the weight information is updated on the condition that image selection has been performed by the switcher 5. As a result, in response to the selection of a certain camera (certain composition) by the switcher 5, it is possible to adjust which composition is made more likely (or less likely) to be selected for the other cameras that were not selected. Therefore, it is possible to make it easier for a composition suitable for the next image selection by the switcher 5 to be selected for the cameras that were not selected, and it is possible to increase the likelihood that captured images with appropriate compositions will be included in the captured images to be selected by the switcher 5, leading to an improvement in the quality of the captured image content.

[0132] In a second example, the weight information is updated when a predetermined sound change is detected from the audio analysis results of the image capture event. For example, the weight information can be updated when a predetermined musical melody change, such as a transition from a vocal part to an instrument solo part, is detected. This makes it possible to update the weight information so that an appropriate composition corresponding to a specific content is more likely to be selected when the sound content of the image capture event transitions to a specific content, such as by updating the weight information so that an appropriate composition corresponding to the specific content is more likely to be selected when a sound change that is estimated to be a transition to a guitar solo part is detected. This can improve the quality of the captured image content.

[0133] For confirmation, an example of the processing procedure to be executed by the CPU 11 to realize semi-automatic control according to the embodiment described above will be described with reference to the flowcharts of Figures 20 and 21. Figure 20 shows an example of the processing procedure for realizing composition control in which a composition is selected based on a composition selection table and the selected composition is switched to. First, in step S11, the CPU 11 waits until a composition switching trigger is generated. As will be understood from the above explanation, in this example, the conditions for generating the composition switching trigger are that an image has been selected by the switcher 5 and that the image is in a musical accompaniment section, and in step S11 the CPU 11 waits for these conditions to be met.

[0134] When a composition switching trigger occurs, the CPU 11 proceeds to step S12, where it performs processing to identify a composition that will result in a prohibited transition for each target camera. That is, based on the prohibited transition composition information described above and information on the composition of the camera currently selected by the switcher 5 (based on the selection history information of the camera described above), it performs processing to identify, as a "prohibited transition composition," a composition that will result in a prohibited transition if selected next by the switcher 5 from among the compositions in the composition selection table of each camera. For confirmation, the identification processing of step S12 may result in an identification result that there is no composition that corresponds to a prohibited transition composition in at least any of the composition selection tables.

[0135] In step S13 following step S12, the CPU 11 performs a process of selecting a composition based on the composition selection table, targeting compositions excluding compositions that result in prohibited transitions. That is, for a composition selection table that has a composition that corresponds to a prohibited transition composition, composition selection is performed based on weight information for compositions excluding the prohibited transition composition. For a composition selection table that does not have a composition that corresponds to a prohibited transition composition, composition selection is performed based on weight information for each composition in the table. Note that if all compositions are prohibited transition compositions, the composition is determined in anticipation of the next composition selection with respect to the selections made by other cameras.

[0136] Here, as described above, when composition selection is performed based on the composition selection table excluding a camera for which an image is being selected by the switcher 5, composition selection based on the composition selection table is not performed for the relevant camera in step S13. This makes it possible to prevent composition switching from being performed for a camera for which an image is being selected by the switcher 5.

[0137] In step S14 following step S13, the CPU 11 controls the composition of the target camera so that the selected composition is achieved. That is, the camera for which composition selection was performed based on the composition selection table in step S13 is set as the target camera, and control is performed so that the composition of the target camera becomes the selected composition.

[0138] In step S15 following step S14, the CPU 11 determines whether a processing termination condition is met. The processing termination condition here is a predetermined condition that has been set in advance as a condition for terminating the series of processing shown in Fig. 20, such as when the generation of captured image content has reached a state where it is necessary to terminate the process.

[0139] If it is determined that the processing termination condition is not satisfied, the CPU 11 returns to step S11. As a result, in response to the occurrence of another composition switching trigger, composition selection based on the composition selection table and switching to the selected composition are executed again for the corresponding camera.

[0140] On the other hand, if it is determined that the processing end condition is met, the CPU 11 ends the series of processing shown in FIG.

[0141] 21 is a flowchart of a process related to updating weight information. In step S21, the CPU 11 waits for a weight update trigger to occur. As will be understood from the above explanation, the weight update trigger can be one of the first and second examples described above. In the first example, in step S21, the CPU 11 waits for an image to be selected by the switcher 5. In the second example, in step S21, the CPU 11 waits for a predetermined sound change to be detected from the audio analysis results of the event to be imaged (for example, a change from a singing part to a solo instrument part to be detected).

[0142] When a weight update trigger occurs, the CPU 11 proceeds to step S22 and performs processing to determine a weight for each composition in the composition selection table of each camera. For example, when updating weights so that compositions that are frequently selected by the switcher 5 are more likely to be selected as described above, a higher value is determined for the weight information of compositions that are selected by the switcher 5 at a certain frequency or more. Furthermore, when updating weights based on the content of the event to be captured, for example, when a solo part of an instrument is detected, a higher value is determined for the weight information of a composition in which the player of that instrument is the subject of image capture.

[0143] In step S23 following step S22, the CPU 11 performs a process of updating the weights to the determined weights, i.e., the CPU 11 updates the numerical values ​​of the corresponding weight information among the weight information in the composition selection table for each camera to the numerical values ​​determined in step S22.

[0144] In step S24 following step S23, the CPU 11 determines whether a processing termination condition is met. The processing termination condition here is a predetermined condition that has been set in advance as a condition for terminating the series of processing shown in Fig. 21, such as when the generation of captured image content has reached a state where it is necessary to terminate the process.

[0145] If it is determined that the processing end condition is not met, the CPU 11 returns to step S21, whereby the weight information is updated again in response to the occurrence of a weight update trigger.

[0146] On the other hand, if it is determined that the processing end condition is met, the CPU 11 ends the series of processing shown in FIG.

[0147] In the above example, composition selection based on the composition selection table is performed on the condition that image selection has been performed by the switcher 5. However, composition selection based on the composition selection table can also be performed in response to changes in the content of the event to be imaged. For example, composition selection based on the composition selection table can be performed on the condition that a predetermined change in musical style (change in event content), such as a change from a vocal part to a solo instrument part, is detected based on the results of audio analysis of the event to be imaged. In this way, by selecting a composition based on the composition selection table in response to changes in the content of the event to be imaged, it is possible to ensure that appropriate composition switching is performed in response to changes in the content of the event to be imaged.

[0148] Furthermore, composition selection based on the composition selection table may be performed depending on the amount of time that has elapsed since the composition was switched based on the previous composition selection. For example, composition selection may be performed on the condition that a certain amount of time has elapsed since the composition was switched based on the previous composition selection.

[0149] It is also conceivable that composition selection based on the composition selection table may be performed in response to a predetermined operation input by the user. In this case, for example, by operating a predetermined button, composition selection based on the composition selection table and switching to the selected composition are performed for cameras other than the camera for which an image is being selected by the switcher 5.

[0150] Furthermore, the weight information can be updated by lowering the weight of compositions that have been selected by the switcher 5 based on the selection history information of the camera by the switcher 5. This updates the weight so that compositions that have been selected by the switcher 5 are less likely to be selected, preventing the same compositions from appearing frequently in captured image content and preventing a decline in content quality.

[0151] An example of the procedure for setting semi-automatic control will be described with reference to Figures 22 and 23. To enable semi-automatic control, as shown in Figure 22, the add controller button Bp1 provided in the controller setting area As3 of the setting screen Gs (in common setting mode) is operated. In response to the operation of the add button Bp1, a dialog box is displayed as shown in the figure, asking whether or not to perform semi-automatic setting. The user can transition to the display of a semi-automatic pattern setting screen as shown in Figure 23 by operating the "Yes" button B7 provided in the dialog box. When the "No" button B8 provided in the dialog box is operated, a regular controller slot is added to the controller setting area As3 for manually setting the execution instruction button Bm for each camera as described above in Figure 8.

[0152] 23, the semi-automatic pattern setting screen has check boxes CBA for selecting the number of members of the artist to be imaged. Specifically, in this example, four check boxes CBA are provided to enable selection of 3-piece, 4-piece, 5-piece, and others. These check boxes CBA are provided to instruct filtering so that the artist personnel composition pattern information displayed on the semi-automatic pattern setting screen is limited to information on the corresponding number of members among the artist personnel composition patterns that can be handled by semi-automatic control.

[0153] On the semi-automatic pattern setting screen, a check box cbb is provided for selecting the displayed personnel composition pattern information. The user checks the check box cbb for selecting the personnel composition pattern of the artist to be photographed from the displayed personnel composition patterns as shown in the figure, and then presses the OK button B9. In response, the information processing device 1 generates a controller linked to a composition selection table determined in accordance with the selected personnel composition pattern.

[0154] Although not shown in the drawings, in response to the operation of the OK button B9, the information processing device 1 displays an operation screen Gm that reflects the controller. Specifically, the information processing device 1 displays the operation screen Gm on which execution instruction buttons Bm representing compositions that can be selected in the composition selection table, such as "vocal up" or "guitar WS," are arranged. During execution of semi-automatic control, when a certain composition is being selected based on the composition selection table, the information processing device 1 performs processing to display the execution instruction button Bm indicating the selected composition on the operation screen Gm in a display mode different from that of the execution instruction button Bm indicating a composition that is not being selected.

[0155] Here, the information processing device 1 switches semi-automatic control ON / OFF in response to the operation of the semi-automatic ON / OFF button B1 on the operation screen Gm.

[0156] It is possible that semi-automatic control is not turned on / off based on user operation, but is automatically turned on / off based on preset conditions. For example, since frequent changes in composition are undesirable during MC, semi-automatic control may be automatically turned off. On the other hand, semi-automatic control may be automatically turned on during performance. Alternatively, semi-automatic control may be automatically turned on / off depending on the frequency of camera selection by the switcher 5. For example, semi-automatic control may be turned on when the frequency of camera selection by the switcher 5 is high, and turned off when the frequency is low.

[0157] Furthermore, if the semi-automatic control is turned off, it is also conceivable that the composition control of the effects, such as "gradual zoom," can also be turned off in conjunction with this.

[0158] (3-4. Regarding the functional configuration related to composition control as an embodiment) Figure 24 is an explanatory diagram of the functions related to composition control as an embodiment possessed by the CPU 11 of the information processing device 1, and shows the parent camera 2, tracking camera 3, pan head 4, switcher 5, and ranging / imaging device 6 shown in Figure 1, along with functional blocks of various functions possessed by the CPU 11.

[0159] As shown in the figure, the CPU 11 has functions as a calibration unit F1, a specification reception unit F2, an operation reception unit F3, a composition selection unit F4, a composition switching control unit F5, a weight update unit F6, an image recognition processing unit F7, an image frame calculation unit F8, a coordinate calculation unit F9, a pan / camera control unit F10, and a cutout image generation unit F11.

[0160] The calibration unit F1 performs a calibration process to identify the same points that appear in both the distance image obtained by the distance measurement / imaging device 6 and the captured image obtained by the tracking camera 3. Note that a specific example of the calibration method used by the calibration unit F1 in this embodiment will be described later.

[0161] The designation receiving unit F2 performs processing for receiving various designations relating to composition control according to the embodiment.

[0162] Specifically, the designation receiving unit F2 performs a process of allowing the user to designate the tracking target subject from among the subjects appearing in the image Ims captured by the distance measurement and image capture device 6. Note that the specific method for accepting the designation of the tracking target subject has already been described with reference to FIGS. 9 and 10 , and therefore a duplicate description will be avoided.

[0163] In addition, as explained above in Figure 12, when the brightness of the captured image Ims becomes equal to or lower than a predetermined brightness, the designation receiving unit F2 performs processing to display an image based on the distance image captured by the distance measuring / capturing device 6 as an image for accepting the designation of the subject to be tracked.

[0164] Furthermore, as described above with reference to FIG. 11, the designation receiving unit F2 performs processing to display name information of the tracking target subject in a position corresponding to the detection frame Fd of the tracking target subject in response to the designation of the tracking target subject.

[0165] Furthermore, as explained above with reference to FIG. 11, in response to the designation of a tracking target subject, the designation receiving unit F2 performs processing to display an image of the designated tracking target subject in the image display area ti of the corresponding subject in the tracking target display area At.

[0166] Furthermore, the designation receiving unit F2 receives a designation of a tracking composition, as described above with reference to FIGS. 13 to 15, and performs processing to calculate the vertical positional deviation amount Vrc and the horizontal offset amount Hof.

[0167] Furthermore, the designation receiving unit F2 receives an instruction to execute designated tracking composition control, which controls the composition of the tracking camera 3 so that the tracking composition is a tracking composition designated by the user for the tracking target subject, and also receives an instruction to execute semi-automatic control. In this example, the reception of an instruction to execute designated tracking composition control corresponds to the reception of an operation of the corresponding execution instruction button Bm arranged on the operation screen Gm. In addition, in this example, the reception of an instruction to execute semi-automatic control corresponds to the reception of an operation of the semi-automatic ON / OFF button B1 described above.

[0168] The operation reception unit F3 receives operations on the operation screen Gm. The operation reception unit F3 recognizes which execution instruction button Bm has been operated for which camera among CAM1 to CAM6.

[0169] The composition selection unit F4, composition switching control unit F5, and weight update unit F6 are functional units for realizing semi-automatic control. The composition selection unit F4 selects a composition for each camera based on the composition selection table described above. The composition switching control unit F5 performs control for switching the composition of each camera to the composition selected by the composition selection unit F4. Specifically, the composition switching control unit F5 performs processing for instructing the image frame calculation unit F8 on the composition selected for each camera by the composition selection unit F4.

[0170] The weight update unit F6 updates the weight information in the composition selection table. The specific method for updating the weight information has already been explained, so a duplicate explanation will be avoided.

[0171] During execution of the designated tracking composition control, the image frame calculation unit F8 calculates a target image frame of the tracking camera 3 for tracking the tracking target subject based on three-dimensional position information of the detection simple model Mbd of the tracking target subject generated based on the distance image obtained by the distance measurement / image capture device 6. Specifically, it performs processing to calculate the vertical center coordinates and horizontal center coordinates of the target image frame based on the three-dimensional position information of the detection simple model Mbd, the vertical position deviation amount Vrc, and the horizontal offset amount Hof. In addition, the image frame calculation unit F8 calculates the target image frame so as to achieve a composition according to the operation content on the operation screen Gm recognized by the operation reception unit F3. For example, it calculates the target composition so as to realize composition control of the effect corresponding to the execution instruction button Bm selected from the execution instruction buttons Bm related to the effect.

[0172] During semi-automatic control, the image frame calculation unit F8 calculates a target image frame so that the composition instructed by the composition switching control unit F5 is realized.

[0173] The coordinate calculation unit F9 converts the information on the target image frame of the tracking camera 3 calculated by the image frame calculation unit F8 (coordinate information in the world coordinate system) into coordinate information in the coordinate system of the tracking camera 3. For this coordinate conversion, a coordinate conversion matrix obtained from the calibration processing result by the calibration unit F1 is used.

[0174] Based on the information on the image frame that has undergone coordinate conversion by the coordinate calculation unit F9, the camera platform / camera control unit F10 controls the pan and tilt of the camera platform 4 and the zoom of the tracking camera 3 as necessary for each tracking camera 3 so as to obtain a composition that captures the range indicated by the information on the image frame. This makes it possible to control the composition of each tracking camera 3 to a target composition.

[0175] The cutout image generation unit F11 performs image cutting as necessary on the image captured by the parent camera 2 in accordance with the image frame information for the virtual camera calculated by the image frame calculation unit F8, and generates captured images for CAM4, CAM5, and CAM6.

[0176] The image recognition processing unit F7 can receive images captured by the parent camera 2 and images captured by each tracking camera 3, and performs image recognition processing on the input images. The image recognition processing here includes at least bone estimation processing for the subject as a person. In this embodiment, the image recognition results of the image recognition processing unit F7 for the images captured by each tracking camera 3, specifically, information on the simple model of the subject generated by the bone estimation processing, are used in the calibration processing by the calibration unit F1.

[0177] 4. Processing Procedures> A specific example of processing procedures to be executed by the CPU 11 to realize the functions related to composition control as the embodiment described above will be described with reference to the flowcharts of FIGS.

[0178] 25 shows an example of a processing procedure relating to a screen display corresponding to a tracking target subject designated by a user. When the processing shown in this figure is executed, it is assumed that an operation screen Gm is displayed on the display unit 17.

[0179] First, in step S101, the CPU 11 waits until an individual display area of ​​any target in the tracking target display area At on the operation screen Gm is operated. If any individual display area is operated, the CPU 11 proceeds to step S102 and determines whether or not the captured image Ims of the distance measurement / image capture device 6 is not yet displayed on the operation screen Gm. If the captured image Ims is not yet displayed, the CPU 11 proceeds to step S103 and executes display processing of the captured image Ims of the distance measurement / image capture device 6. For example, processing is performed to superimpose the captured image Ims of each distance measurement / image capture device 6 on the performance area Are on the operation screen Gm. After executing the display processing of step S103, the CPU 11 proceeds to step S104.

[0180] On the other hand, if it is determined in step S102 that the captured image Ims is not undisplayed, the CPU 11 skips the display process of step S103 and proceeds to step S104. In other words, if the captured image Ims is already displayed on the operation screen Gm in response to an operation of designating a tracking target subject, for example, the display process of step S103 is not executed.

[0181] In step S104, the CPU 11 waits for an operation on the detection frame Fd (an operation to select the detection frame Fd), and if an operation on the detection frame Fd has been performed, the CPU 11 proceeds to step S105, where it performs a process of associating the subject of the operated detection frame Fd with the operated target. That is, it performs a process of associating the subject appearing within the operated detection frame Fd with the subject whose individual display area was operated in step S101.

[0182] In step S106 following step S105, the CPU 11 performs a process of displaying object name information for the detection frame Fd, i.e., a process of displaying the name information set for the object whose individual display area was operated in step S101 at a position corresponding to the detection frame Fd in which the object whose detection frame Fd was operated in step S104 appears.

[0183] In step S107 following step S106, the CPU 11 performs processing to display a captured image of the designated subject in the corresponding image display area ti of the tracking target display area At. That is, the CPU 11 performs processing to display a captured image of the subject designated as the tracking target subject (an image captured by the distance measurement / image capture device 6), specifically, an image extracted from the captured image Ims, in the image display area ti in the individual display area operated in step S101.

[0184] In step S108 following step S107, the CPU 11 determines whether or not the processing has ended, that is, whether or not a predetermined condition has been established that indicates that the series of processing steps shown in Fig. 25 should end. If it is determined in step S108 that the processing has not ended, the CPU 11 returns to step S101. This makes it possible to accept the designation of another tracking target subject, and to display name information of the designated subject in the detection frame Fd when a tracking target subject has been designated.

[0185] If it is determined in step S108 that the process has ended, the CPU 11 ends the series of processes shown in FIG.

[0186] 26 is a flowchart of a process for switching the display depending on the brightness of the captured image Ims displayed on the operation screen Gm. In step S201, the CPU 11 executes a brightness detection process for the captured image Ims of the distance measurement / image capture device 6. For example, the CPU 11 executes a process for detecting the average brightness value of the captured image Ims.

[0187] In step S202 following step S201, the CPU 11 determines whether the brightness is equal to or less than a predetermined value, specifically, for example, whether the average brightness value is equal to or less than a predetermined value. For example, if it is determined that the average brightness value is equal to or less than the predetermined value and the brightness is equal to or less than the predetermined value, the CPU 11 proceeds to step S203 to determine whether a distance image is being superimposed. That is, it determines whether a distance image captured by the distance measurement / image capture device 6 is being superimposed on the captured image Ims. If a distance image is not being superimposed, the CPU 11 proceeds to step S204 to perform a distance image superimposition process, i.e., a process of superimposing a distance image captured by the distance measurement / image capture device 6 on the captured image Ims being displayed, and then proceeds to step S207.

[0188] On the other hand, if the CPU 11 determines in step S203 that the distance image is not being superimposed, it skips the superimposition process in step S204 and proceeds to step S207.

[0189] If the CPU 11 determines in step S202 that the brightness is not equal to or less than the predetermined value, the process proceeds to step S205 to determine whether a distance image is being superimposed, and if so, the process proceeds to step S206 to stop the superimposition, i.e., to stop the superimposed display of the distance image, and then the process proceeds to step S207. On the other hand, if the CPU 11 determines in step S205 that a distance image is not being superimposed, the process skips the superimposition stop process of step S206 and proceeds to step S207.

[0190] In step S207, the CPU 11 determines whether or not the processing has ended. If it is determined that the processing has not ended, the process returns to step S201, and if it is determined that the processing has ended, the series of processing steps shown in FIG. 26 ends.

[0191] 27 is a flowchart of the process to be executed in response to the user's designation of the tracking composition. In step S301, the CPU 11 waits until the designation of the tracking composition is completed. Specifically, in this example, the CPU 11 waits until the end button B6 on the composition designation screen shown in FIG. 13 is operated.

[0192] If it is determined that the end button B6 has been operated and the specification of the tracking composition has been completed, the CPU 11 proceeds to step S302, where it performs a process of calculating the intersection point between the vertical center line of the image frame Fs and the center line of the virtual simple model. That is, it performs a process of calculating the intersection point between the vertical center line Vc of the image frame Fs and the horizontal center line of the virtual simple model Mbv, which is the intersection point J described with reference to FIG. 14B .

[0193] In step S303 following step S302, the CPU 11 calculates the vertical positional deviation Vrc based on the height of the virtual simple model Mbv. That is, the vertical position of the intersection J is scaled by the first vertical scaler to calculate the vertical positional deviation Vrc.

[0194] In step S304 following step S303, the CPU 11 calculates the horizontal offset amount Hof of the intersection point J based on the width of the image frame Fs. That is, the horizontal offset amount Hof is calculated as a value obtained by scaling the horizontal position of the intersection point J (the horizontal position of the horizontal center line of the virtual simple model Mbv) using the horizontal scaler described above.

[0195] In step S305 following step S304, the CPU 11 performs a process of storing information on the zoom amount Zr obtained from the zoom operation when specifying the composition, the vertical position shift amount Vrc, and the horizontal offset amount Hof, for example, in a predetermined storage device such as the memory unit 19.

[0196] After executing the process of step S305, the CPU 11 ends the series of processes shown in FIG.

[0197] 28 is a flowchart of a process related to calculation of a target image frame for realizing a specified tracking composition. In step S401, the CPU 11 performs a process of acquiring three-dimensional coordinate information of a simple detection model Mbd of a tracking target. Specifically, in this example, the CPU 11 performs a process of acquiring three-dimensional coordinate information of the simple detection model Mbd of a tracking target subject from the distance measurement / image capture device 6.

[0198] In step S402 following step S401, the CPU 11 executes a case determination process, that is, determines which of the above-mentioned cases 1 to 4 applies based on the stored value of the vertical positional deviation amount Vrc.

[0199] If the result is case 1, the CPU 11 proceeds to step S403, executes processing to calculate the center coordinates of the target image frame according to case 1, and proceeds to step S407. If the result is case 2, the CPU 11 proceeds to step S404, executes processing to calculate the center coordinates of the target image frame according to case 2, and proceeds to step S407. If the result is case 3, the CPU 11 proceeds to step S405, executes processing to calculate the center coordinates of the target image frame according to case 3, and proceeds to step S407. If the result is case 4, the CPU 11 proceeds to step S406, executes processing to calculate the center coordinates of the target image frame according to case 4, and proceeds to step S406. Note that specific examples of the processing to calculate the center coordinates of the target image frame according to each of cases 1 to 4 have already been described, so repeated description will be avoided.

[0200] In step S407, the CPU 11 performs processing to control the pan, tilt, and zoom of the target camera based on the calculated center coordinates of the target image frame and the stored information on the zoom amount Zr.

[0201] After executing the process of step S407, the CPU 11 ends the series of processes shown in FIG.

[0202] 5. Calibration> The calibration process by the calibration unit F1 will be described. As mentioned above, the calibration process in this embodiment is a process for understanding the positional relationship between each tracking camera 3 and each ranging / image capturing device 6. Specifically, it is a process for identifying the same points that appear in both the distance image obtained by the ranging / image capturing device 6 and the captured image obtained by the tracking camera 3, between each ranging / image capturing device 6 and each tracking camera 3.

[0203] When the subject of imaging is, for example, a live music event, the calibration process often has to be carried out while various preparations for the event are being carried out on stage. In such cases, if calibration is carried out using a conventional test board, the stage must be occupied for calibration, which causes a delay in the preparation work, which is undesirable.

[0204] In view of these circumstances, in this embodiment, a calibration process is performed to identify the positions of identical parts between a simple model on the sensor device side, which is a simple model of the subject generated from a distance image obtained by the distance measurement / imaging device 6, and a simple model on the camera side, which is a simple model of the subject generated from an image obtained by the tracking camera 3.

[0205] Specifically, as shown in Fig. 29, a person 50 is placed on a stage as a subject for calibration, and a calibration process is performed to identify the positions of identical parts between the sensor device side simple model and the camera side simple model of this person 50. As shown in Fig. 29, the coordinates of a certain part of the sensor device side simple model of the person 50 are expressed as three-dimensional space coordinates of XYZ = P N (x N , y N , z N ), whereas the coordinates of the same part of the camera-side simple model of the person 50 are expressed by coordinates = Q in the camera coordinate system (uv coordinate system). N (u N , v N In this case, the calibration process may be performed by identifying the three-dimensional position of a certain part of the person 50 detected on the sensor device side simple model and the uv coordinate system position detected on the camera side simple model.

[0206] In this case, during calibration, the tracking function of the person 50 allows the person 50 to move around on the stage, so that performing the calibration does not interfere with preparations for the start of the event.

[0207] In this embodiment, a first method and a second method are proposed as calibration processing methods. Fig. 30 is an explanatory diagram of the calibration processing method as the first method. In the first method, data sampling for calibration is performed while switching between the target distance measurement / image capture devices 6. In this first method, data sampling for calibration is performed the same number of times as the number of distance measurement / image capture devices 6.

[0208] A specific example of the procedure for performing calibration is shown in the flowchart of Fig. 31. First, in step S51, the user selects one distance measuring / image capturing device 6 to be calibrated. In the following step S52, the user adjusts the pan, tilt, and zoom of all cameras (all tracking cameras 3) so that the distance measuring range of the selected distance measuring / image capturing device 6 is captured.

[0209] Next, in step S53, the user issues an instruction to start sampling processing for calibration to the information processing device 1. The sampling processing for calibration referred to here means processing for acquiring, as sample data for calibration, coordinate information (three-dimensional coordinates) of a predetermined part of the sensor device side simple model generated for the person 50 on the stage and coordinate information (uv coordinates) of a predetermined part of the camera side simple model generated for the person 50.

[0210] In step S54 following step S53, person 50, who is the subject of calibration, raises both hands while moving within the selected ranging range of distance measurement / image capture device 6. By raising both hands, person 50 can recognize person 50 on the information processing device 1 side even if there is a person other than person 50 on the stage.

[0211] Then, in step S55 following step S54, it is determined whether or not sufficient samples have been obtained. If sufficient samples have not been obtained, the movement of person 50 in step S54 continues. If sufficient samples have been obtained, in step S56 the user instructs information processing device 1 to end the calibration sampling process.

[0212] In the first technique, the procedure shown in FIG. 31 is performed for each distance measuring and image capturing device 6 while switching the target distance measuring and image capturing device 6 .

[0213] 32 is a flowchart of processing executed by the CPU 11 of the information processing device 1 when the first method is adopted. In step S501, the CPU 11 waits for the start of sampling processing. Specifically, the CPU 11 waits for an instruction to start the sampling processing for calibration performed in step S53 in FIG.

[0214] If it is determined that the sampling process has started, the CPU 11 proceeds to step S502 and acquires simple model information obtained by the distance measurement / image capture device 6. That is, information on the sensor device side simple model obtained by the distance measurement / image capture device 6 selected as the target (for example, coordinate information of a specific part of the sensor device side simple model) is acquired.

[0215] In step S503 following step S502, the CPU 11 extracts and stores simple model information of the calibration subject. That is, from the acquired information on the sensor device side simple model, only information on the person 50 with both hands raised is extracted and stored in a predetermined storage device such as the storage unit 19.

[0216] In step S505 following step S504, the CPU 11 performs image recognition processing on the images captured by each camera (each tracking camera 3), and in the subsequent step S505, it extracts and stores simple model information of the calibration subject (i.e., person 50) from the image recognition processing results of each camera.

[0217] In step S506 following step S505, the CPU 11 determines whether the sampling process has ended, that is, whether an instruction to end the calibration sampling process performed in step S56 in Fig. 31 has been issued. If it is determined that the sampling process has not ended, the CPU 11 returns to step S502. That is, in this case, the calibration sampling process continues.

[0218] On the other hand, if it is determined that the sampling process has ended, the CPU 11 proceeds to step S507, and performs a process of identifying identical points between the distance measurement / imaging device 6 and each camera based on the stored simple model information, and then completes the series of processes shown in Figure 32.

[0219] The processing of the CPU 11 corresponding to the first technique described with reference to Fig. 32 can be expressed as follows: That is, the unit sampling process for sampling the positions of parts of the sensor device-side simple model and the camera-side simple model is executed for each distance measurement / image capture device 6 while the target distance measurement / image capture device 6 is switched, with the composition adjusted so that each tracking camera 3 captures the distance measurement range of one target distance measurement / image capture device 6.

[0220] By performing the above-mentioned unit sampling process for each ranging / imaging device 6, it is possible to acquire sample data necessary for calibration processing (identification of the same point) between each tracking camera 3 and each ranging / imaging device 6. In each unit sampling process, it is possible to make the imaging range of each tracking camera 3 sufficiently overlap with the ranging range of the target ranging / imaging device 6, so that calibration processing can be directly performed for each tracking camera 3 with the target ranging / imaging device 6. Therefore, by performing the unit sampling process for each ranging / imaging device 6 while switching the target ranging / imaging device 6 as described above, it is possible to perform calibration processing between each tracking camera 3 and each ranging / imaging device 6 with high accuracy.

[0221] FIG. 33 is an explanatory diagram of a calibration processing technique as a second technique. The second technique is a technique that enables the necessary number of calibration sampling processes to be limited to one. If the imaging ranges of all tracking cameras 3 can be made to sufficiently overlap with the ranging ranges of all ranging / imaging devices 6, the number of calibration sampling processes can be limited to one. However, according to the arrangement of the tracking cameras 3 and ranging / imaging devices 6 in this example, when tracking camera 3-1 attempts to make its own imaging range sufficiently overlap with the ranging ranges of ranging / imaging devices 6-2 and 6-4 arranged on the opposite side (upstream side), it becomes unable to make its own imaging range sufficiently overlap with the ranging ranges of ranging / imaging devices 6-1 and 6-3 arranged on the same side (downstream side). Similarly, for the tracking camera 3-2, if the tracking camera 3-2 attempts to make its own imaging range sufficiently overlap with the ranging ranges of the ranging / imaging devices 6-1 and 6-3 arranged on the opposite side (downstream side), it will not be able to make its own imaging range sufficiently overlap with the ranging ranges of the ranging / imaging devices 6-2 and 6-4 arranged on the same side (upstream side).It is not possible to directly identify the same point between the tracking camera 3 and ranging / imaging devices 6, which are in a relationship where the imaging range cannot be sufficiently overlapped with the ranging range. Therefore, in this case, it is possible to perform calibration sampling processing separately for each of the following states: a state in which the tracking camera 3-1 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-2 and 6-4 and the tracking camera 3-2 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-1 and 6-3; and a state in which the tracking camera 3-1 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-1 and 6-3 and the tracking camera 3-2 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-2 and 6-4.

[0222] However, in this example, the tracking camera 3-3 placed at the FOH is capable of capturing the entire stage, and therefore its own imaging range can be made to sufficiently overlap with the ranging ranges of all ranging / imaging devices 6 and the imaging ranges of all other tracking cameras 3 (i.e., 3-1 and 3-2).

[0223] Therefore, in the second method, the calibration process between the tracking camera 3 and the ranging / imaging device 6, which is a relationship in which the imaging range cannot be sufficiently overlapped with the ranging range, is performed along the tracking camera 3-3 arranged in the HOF, thereby reducing the number of calibration sampling processes required to one.

[0224] FIG. 34 is a flowchart showing a specific example of the calibration execution procedure in the second method. In this case, in step S61, the user adjusts the pan, tilt, and zoom of each camera (each tracking camera 3) as follows. That is, FOH (tracking camera 3-3): Captures the entire stage (enough overlap between the ranging ranges of all ranging / imaging devices 6, the imaging range of tracking camera 3-1, and the imaging range of tracking camera 3-2). Downstream (tracking camera 3-1): Captures the ranging range of the ranging / imaging devices 6 (6-2 and 6-4) installed on the upstream side. Upstream (tracking camera 3-2): Captures the ranging range of the ranging / imaging devices 6 (6-1 and 6-3) installed on the downstream side. Next, in step S62, the user issues an instruction to the information processing device 1 to start calibration sampling processing.

[0225] In step S63 following step S62, the person 50 who is the subject of the calibration moves through the distance measurement ranges of all distance measurement / image capture devices 6 while raising both hands.

[0226] Then, in step S64 following step S63, it is determined whether or not sufficient samples have been obtained. If sufficient samples have not been obtained, the movement of person 50 in step S63 continues. If sufficient samples have been obtained, in step S65 the user instructs information processing device 1 to end the calibration sampling process.

[0227] In this case, the CPU 11 performs the same calibration processing as that shown in FIG. 32. However, in this case, the CPU 11 performs the identification processing of the identical points in step S507 along the tracking camera 3-3 when the identification processing between the tracking camera 3 and the distance measurement / image capture device 6, which are in a relationship where the imaging range cannot sufficiently overlap with the ranging range, is performed along the tracking camera 3-3. Specifically, the identification processing of the identical points between the tracking camera 3-1 and the distance measurement / image capture devices 6-1 and 6-3 is performed based on the identification results of the identical points between the tracking camera 3-1 and the tracking camera 3-3 and the results of the calibration processing between the tracking camera 3-3 and the distance measurement / image capture devices 6-1 and 6-3. Furthermore, the identification processing of the identical points between the tracking camera 3-2 and the distance measurement / image capture devices 6-2 and 6-4 is performed based on the identification results of the identical points between the tracking camera 3-2 and the tracking camera 3-3 and the results of the calibration processing between the tracking camera 3-3 and the distance measurement / image capture devices 6-2 and 6-4.

[0228] By adopting the second method described above, it is no longer necessary to perform separate calibration sampling processes for each of the following states: a state in which the tracking camera 3-1 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-2 and 6-4 and the tracking camera 3-2 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-1 and 6-3; and a state in which the tracking camera 3-1 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-1 and 6-3 and the tracking camera 3-2 sufficiently overlaps its own imaging range with the ranging range of the ranging / imaging devices 6-2 and 6-4; and the number of required calibration sampling processes can be reduced to one.

[0229] Here, the calibration processing method as the second method as described above can be expressed as follows: That is, there are a plurality of distance measuring / imaging devices 6 and a plurality of tracking cameras 3, the tracking cameras 3 include a first tracking camera (tracking camera 3-3) arranged so as to be able to capture the imaging ranges of all the other tracking cameras 3 and the distance measuring ranges of all the distance measuring / imaging devices 6, and a second tracking camera (tracking camera 3-1 or 3-2) arranged on either the left or right side as viewed from the first tracking camera so as to be able to capture the imaging range of the first tracking camera, the distance measuring / imaging device 6 includes a same-side sensor device and an opposite-side sensor device arranged at a position on the same side as the second tracking camera and a position on the opposite side as viewed from the first tracking camera, and the calibration unit F1 performs calibration processing between the first tracking camera and the same-side sensor device based on the result of identifying the same point between the second tracking camera and the first tracking camera and the result of calibration processing between the first tracking camera and the same-side sensor device.

[0230] If calibration processing between each tracking camera 3 and each ranging / imaging device 6 is to be performed in a single sampling process, the imaging range of each tracking camera 3 needs to be sufficiently overlapped with the ranging ranges of all ranging / imaging devices 6. However, if the imaging range of the second tracking camera is to be sufficiently overlapped with the ranging range of the opposite-side sensor device, it may not be possible to sufficiently capture the ranging range of the same-side sensor device, in which case calibration processing between the second tracking camera and the same-side sensor device cannot be performed directly. According to the processing of the calibration unit F1 described above, even if calibration processing between the second tracking camera and the same-side sensor device cannot be performed directly, calibration processing can be performed along the first tracking camera. In other words, when realizing the calibration processes between the second tracking camera and the opposite-side sensor device, and between the second tracking camera and the same-side sensor device, there is no need to perform separate sampling processes for calibration in a state where the imaging range of the second tracking camera is sufficiently overlapped with the ranging range of the opposite-side sensor device, and in a state where the imaging range of the second tracking camera is sufficiently overlapped with the ranging range of the same-side sensor device, and the calibration processes between each tracking camera and each sensor device can be realized with a single sampling process.

[0231] In addition, the calibration process can also allow the user to specify on the image a position on an object on the stage that will be the same point, as shown in the example of Fig. 35. In this case, when specifying a position on an image captured by the distance measurement / image capture device 6 that will be the same point, it is also possible to detect brightness of the captured image, and if the brightness is below a predetermined value, to display a distance image superimposed on it. This allows the user to properly specify the same point even when the stage is dark.

[0232] 6. Modifications Here, the embodiment is not limited to the specific examples described above, and various modified configurations may be adopted. For example, in the description above, the tracking composition is a composition for tracking a single subject, but it is also possible to designate a tracking composition that includes multiple subjects, in other words, a composition for a group shot.

[0233] 36A shows an example of a group shot setting reception screen for receiving a designation of a tracking composition for such a group shot. As shown in FIG. 36A, the group shot setting reception screen may first display a screen on which check boxes cbg are arranged for designating a tracking target for the group shot, with the target being a subject whose name information has been set in the subject name setting area As2 (see FIGS. 6 and 7 ) described above. The user can select a subject whose check box cbg is checked as a tracking subject to be included in the group shot, and can confirm the selection of the tracking target subject by operating the OK button B11 in the figure.

[0234] When the OK button B11 is operated, the screen transitions to a screen on which virtual simple models Mbv of multiple subjects designated as tracking targets are arranged, as shown in Fig. 36B . On this screen, the user performs an operation to select a part that the user wants to include in the composition from among the parts displayed on the virtual simple model Mbv, and then operates the end button B12.

[0235] In this case, the target image frame can be calculated so that all of the parts specified in the group shot settings in the detection simple models Mbd are included within the image frame, based on information about the detection simple models Mbd for the subjects specified as tracking targets for the group shot, obtained by the distance measurement / image capture device 6. As an example, the image frame can be calculated with a size that allows all of the specified parts to fit within the image frame, with the position at the center of the image frame being the average value of the minimum and maximum values ​​on each axis (e.g., X, Y, and Z axes) of the specified parts in each detection simple model Mbd.

[0236] The center of the target image frame may be set to a position offset by a specified amount in the horizontal and vertical directions from the position of the average value. Also, the image frame size may be set to a size that barely includes all the specified parts as a reference size, and a size offset by a specified amount from that size may be set.

[0237] Regarding composition control for group shots, it is conceivable to allow the user to specify a main person, and exclude people who are further away from the main person than a specified distance from the main person from being tracked. It is also conceivable to configure the system so that only people who are within a pre-specified range in three-dimensional space are tracked.

[0238] 37 shows an example of an execution instruction button Bm ("vocals, guitar" in the figure) for instructing a tracking composition for a group shot. A tracking indicator It can also be provided for the execution instruction button Bm for instructing a tracking composition for a group shot. In this case, the tracking indicator It can be displayed in different modes depending on whether all subjects set as tracking targets have been detected, only some of the subjects have been detected, or none of the subjects have been detected. For example, the tracking indicator It can be displayed in blue when all subjects have been detected, in yellow when only some of the subjects have been detected, or in gray when none of the subjects have been detected.

[0239] It is also conceivable that the tracking sensitivity for tracking a subject can be variably set. FIG. 38 shows an example in which a tracking sensitivity setting operation unit is provided in, for example, the performance area Are on the operation screen Gm, as indicated by "K" in the figure. In this case, the user can issue a sensitivity switching instruction even while the tracking target subject is being tracked. Therefore, in this case, the CPU 11 controls the switching of the tracking sensitivity while the tracking camera 3 is tracking the tracking target subject. In this case, it is conceivable that the tracking sensitivity can be switched between three levels, for example, low, medium, and high.

[0240] For example, it is possible to adjust the tracking sensitivity appropriately depending on the situation, such as lowering the sensitivity during MC, increasing the sensitivity for intense songs, and lowering the sensitivity for calm songs. It is also conceivable that the sensitivity switching could be automated. In this case, one possible automation would be to turn off semi-automatic control (because it can be assumed that an MC is in progress) and then switch to lowering the sensitivity.

[0241] It is also conceivable that, even while a tracking target subject is being tracked, the composition can be adjusted in response to a manual operation input using, for example, a joystick 30 provided as part of the input unit 16. Fig. 39 shows an example in which a manual adjustment button Ba for instructing switching to such manual adjustment is arranged in, for example, the performance area Are on the operation screen Gm. For example, by turning on the manual adjustment button, the composition controlled in accordance with the selected execution instruction button Bm can be adjusted by a manual operation input using the joystick 30 or the like.

[0242] At this time, manual operation input using the joystick 30 or the like is accepted by the camera control protocol for the virtual camera possessed by the information processing device 1 (CPU 11), as shown in FIG. 40, and a control signal based on the manual operation input is input via the switching unit in the figure to the camera control protocol for the relevant camera among the camera control protocols for CAM1, CAM2, and CAM3, so that the composition of the relevant camera is controlled to the composition based on the manual operation input.

[0243] 41, information indicating the percentage of each tracking target subject selected for PGM out can also be displayed on the operation screen Gm (see "R" in the figure). In this case, the information indicating the percentage can be displayed as statistical information for the entire live performance or for each song. It is also desirable that the information indicating the percentage can be reset in response to a predetermined operation such as operating a reset button.

[0244] In the embodiment, when a tracking target subject is specified, the captured images Ims are displayed on the operation screen Gm (see FIG. 10 etc.), but it is also possible to change the order of the captured images Ims on the screen. This makes it possible to change the order of the captured images Ims on the screen in accordance with the change in the order, for example, when the order of the distance measurement / imaging devices 6 is changed.

[0245] Furthermore, in the above description, an example has been given in which the switcher 5 is configured as a hardware device, but the switcher 5 may also be realized by a software program executed by the information processing device 1 .

[0246] Furthermore, although the explanation so far has exemplified the case where the event to be imaged is a live music event, this technology can also be suitably applied to other events to be imaged, such as musicals and other events where performances are performed on a stage (either indoors or outdoors), program recordings in a studio, and sports events such as baseball, soccer, basketball, and volleyball.

[0247] 7. Recording Medium> The information processing device (same as 1) has been described above as an embodiment, and the recording medium of the embodiment is a recording medium on which a program that causes a computer device such as a CPU to execute processing as the information processing device 1 is recorded.

[0248] The recording medium of the embodiment is a recording medium on which a computer-readable program is recorded, and on which a program that causes the computer to realize a designation reception processing function that performs processing to have a user designate a tracking target subject by a tracking camera capable of tracking a subject from subjects captured in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera. In other words, this recording medium corresponds to a recording medium on which a program that causes the computer to execute the processing described in FIG. 25 etc. is recorded.

[0249] In this case, the program can be stored in advance on a computer-readable recording medium, such as a ROM, HDD (Hard Disk Drive), or SSD (Solid State Drive). Alternatively, the program can be temporarily or permanently stored on a removable recording medium, such as a semiconductor memory, a memory card, an optical disk, a magneto-optical disk, or a magnetic disk. Such removable recording medium can be provided as a so-called package software. In addition, the program can be installed from the removable recording medium onto a personal computer or the like, or can be downloaded from a download site to a required information processing device, such as a smartphone, via a network, such as a LAN or the Internet.

[0250] 8. Summary of the Embodiment As described above, the information processing device (same as 1) according to the embodiment includes a designation reception processing unit (designation reception unit F2) that allows a user to designate a tracking target subject from a subject captured in a sensing image captured by a sensor device having a light-receiving sensor separate from the tracking camera. According to the above configuration, even if the subject the user wants to track is located outside the imaging range of the tracking camera, the sensor device separate from the tracking camera can capture the subject in the sensing image, allowing the user to designate the subject they want to track from the sensing image captured by the sensor device. Furthermore, if a sensor device having a light-receiving sensor capable of receiving infrared light is used, even if the user cannot identify the subject in the image captured by the tracking camera in a dark environment, the subject can be identified in the sensing image captured by the sensor device, making it easier to designate the tracking target subject. Thus, the information processing device according to the embodiment allows the tracking camera to appropriately designate the tracking target subject.

[0251] In addition, in an information processing device according to an embodiment, the sensor device includes a light receiving sensor, which includes a visible light sensor capable of receiving visible light and an infrared light sensor capable of receiving infrared light. The sensor device also includes an imaging function for obtaining an image of visible light based on a light receiving signal from the visible light sensor, and a distance measurement function for obtaining a distance image based on the light receiving signal from the infrared light sensor. The designation acceptance processing unit displays an image based on the distance image as an image for accepting the designation of the tracking target subject when the brightness of the image captured by the sensor device drops below a predetermined brightness level while the image captured by the sensor device is displayed as an image for accepting the designation of the tracking target subject. As a result, when the ambient light around the subject becomes dark and it becomes difficult for the user to identify the subject in the image captured by the sensor device, an image based on the distance image captured by the sensor device, specifically an image of the distance image itself or a simple model of the subject generated based on the distance image, is displayed as an image for accepting the designation of the tracking target subject. This makes it easier to designate the tracking target subject even in dark environments. Therefore, the designation of the tracking target can be performed appropriately even in dark environments.

[0252] Furthermore, in the information processing device as an embodiment, in response to a tracking target subject being designated from a subject appearing in a sensing image captured by the sensor device, the designation reception processing unit performs processing to display name information of the tracking target subject at a position corresponding to the detection frame of the tracking target subject superimposed on the display image of the sensing image. This allows the user to intuitively understand whether the tracking target subject has been designated as intended, based on whether or not the name information of the tracking target subject is displayed relative to the detection frame of the tracking target subject.

[0253] Furthermore, in the information processing device as an embodiment, the designation reception processing unit displays a sensing image from the sensor device on an operation screen on which an instruction operation can be made as to whether or not to track the tracking target subject using at least the tracking camera, and in response to a tracking target subject being designated from a subject appearing in the sensing image, performs processing to display an image of the designated tracking target subject extracted from the sensing image in a tracking target display area provided on the operation screen that displays name information of the tracking target subject.This allows the user to intuitively understand whether or not the designation of the tracking target subject has been performed as intended, based on whether or not the sensing image of the tracking target subject is displayed in the tracking target display area provided on the operation screen.

[0254] Furthermore, in the information processing device as an embodiment, the specification acceptance processing unit accepts a specification of a tracking composition of a subject by a tracking camera as a specification of a tracking image frame for a displayed image of a virtual simple model, which is a virtual simple model of the subject. This makes it possible to specify a tracking composition with a high degree of freedom, allowing the tracking target subject to be positioned at any position within the image frame. Furthermore, according to the above configuration, the specification of the tracking composition is accepted as a specification of a tracking image frame for a visually displayed virtual simple model of the subject, making it easier for the user to imagine the composition of the captured image obtained by tracking, and improving the ease of specifying the tracking composition.

[0255] Furthermore, in the information processing device according to the embodiment, the specification receiving processing unit receives a specification of a tracking composition of a subject by a tracking camera as a specification of the position of a tracking image frame relative to a displayed image of a virtual simple model and the size of the tracking image frame. As a result, when specifying a tracking composition of a subject, the user is only required to perform two operations: a specification operation of the position of the tracking image frame relative to the virtual simple model and a specification operation of the size of the tracking image frame. Therefore, it is possible to reduce the operational burden on the user when specifying a tracking composition of a subject.

[0256] Furthermore, in an information processing device according to an embodiment, the sensor device has an infrared light sensor capable of receiving infrared light as a light receiving sensor, and has a distance measurement function for obtaining a distance image based on the light receiving signal of the infrared light sensor. The sensor device also has an image frame calculation unit (F8) for calculating a target image frame of the tracking camera for tracking the tracking subject based on three-dimensional position information of a detection simple model, which is a simple model of the tracking subject, generated based on the distance image obtained by the sensor device. The specification reception processing unit stores information indicating the positional deviation of the image frame center of the tracking image frame relative to the virtual simple model as relative-model image frame center deviation information. The image frame calculation unit calculates the target image frame so that the positional deviation indicated by the relative-model image frame center deviation information is reproduced as the positional deviation of the image frame center of the target image frame relative to the detection simple model. In this way, by focusing on the positional deviation of the image frame center of the specified tracking image frame relative to the virtual simple model and reproducing the positional deviation as the positional deviation of the image frame centers of the detection simple model and the target image frame, the tracking camera can be appropriately controlled so that the tracking composition matches the specified composition.

[0257] In addition, in an information processing device according to an embodiment, the specification acceptance processing unit calculates the vertical positional deviation amount, which indicates the vertical positional deviation amount included in the image frame center position deviation information relative to the model, as a relative value scaled by a first scaler using the vertical position of a predetermined part of the virtual simple model as a reference value and the length between the predetermined parts spaced apart in the vertical direction on the virtual simple model as a reference unit. The image frame calculation unit determines the vertical center position of the target image frame at the vertical position specified by the vertical positional deviation amount using a second scaler using the vertical position of a predetermined part of the detected simple model as a reference value and the length between the predetermined parts spaced apart in the vertical direction on the detected simple model as a reference unit. This allows the specified tracking composition to be properly reproduced even if the size of the virtual simple model used to specify the tracking composition differs from the size of the actual detected simple model (the size of the tracking target subject). In other words, the specified tracking composition can be properly reproduced regardless of the size of the tracking target subject.

[0258] Furthermore, in the information processing device as an embodiment, the sensor device has, as light receiving sensors, a visible light sensor capable of receiving visible light and an infrared light sensor capable of receiving infrared light, and has an imaging function for obtaining an image of visible light based on a light receiving signal of the visible light sensor, and a distance measuring function for obtaining a distance image based on the light receiving signal of the infrared light sensor, and is provided with an image frame calculation unit for calculating a target image frame of the tracking camera for tracking the tracking target subject based on three-dimensional position information of a detection simple model which is a simple model of the tracking target subject generated based on the distance image obtained by the sensor device, and The processing unit performs processing to display a designation acceptance screen on which multiple virtual simple models, which are virtual simple models of the subjects, are arranged as a designation acceptance screen for tracking target subjects during group tracking in which multiple subjects are tracking target subjects of the tracking camera, and accepts designation of body parts from each of the multiple arranged virtual simple models. The image frame calculation unit, when a detected simple model of each subject set as a tracking target for group tracking is obtained, calculates a target image frame so that the detected body parts of each subject set as a tracking target, designated on the designation acceptance screen, are included within the target image frame. This allows the user to specify a tracking composition during group tracking simply by specifying the body parts they want to include in the tracking composition from at least the multiple arranged virtual simple models. This facilitates the designation of a tracking composition during group tracking. It also facilitates the designation of a tracking composition that includes important body parts of each of the multiple subjects set as tracking targets.

[0259] Furthermore, in an information processing device according to an embodiment, the tracking camera is configured to be able to switch the tracking sensitivity of the tracking camera while the tracking camera is tracking a tracking target subject. The state of the subject may change during the progress of an event to be imaged. For example, at a live music concert, the state of the subject may change in various ways, such as when the subject is MCing, singing or playing an intense song, or singing or playing a calm song. In this case, it may be necessary to switch the tracking sensitivity depending on the state of the subject, for example, while it is desirable to avoid sudden changes in the tracking composition during the MC, while it is desirable to suddenly change the tracking composition during the singing or playing of an up-tempo song. With the above configuration, the tracking sensitivity can be switched while the tracking camera is tracking the subject during the progress of the event to be imaged, thereby realizing appropriate tracking sensitivity control in response to changes in the state of the subject during the progress of the event to be imaged.

[0260] Furthermore, the information processing device according to the embodiment is configured to execute a designated tracking composition control, which controls the composition of the tracking camera to achieve a tracking composition designated by a user for a tracking target subject, and a semi-automatic composition control, which selects a composition for the tracking camera based on a composition selection table in which weight information is associated with each combination of a subject captured by the tracking camera and a tracking camera composition type, and controls the composition of the tracking camera to achieve the selected composition. The designation reception processing unit receives an instruction to execute the designated tracking composition control and an instruction to execute the semi-automatic composition control. This allows the tracking camera to execute not only the designated tracking composition control but also semi-automatic composition control in response to a user's instruction. For example, when capturing image content for an event such as a live music concert, it is desirable to create high-quality content that is not boring by appropriately switching the camera composition. However, selecting an appropriate composition can be difficult for inexperienced users, and this may require the work of an experienced user. Furthermore, manually selecting a composition increases the cost of creating captured image content. According to the semi-automatic composition control, the composition of the tracking camera is automatically switched based on the composition selection table, and the composition switching mode of the tracking camera can be appropriately set by setting the weight information in the composition selection table. Therefore, for captured image content that requires composition switching, it is possible to improve the quality of the content while reducing the work costs involved in creating the content.

[0261] Furthermore, in the information processing device according to the embodiment, the sensor device includes an infrared light sensor capable of receiving infrared light as a light receiving sensor, and a distance measurement function for obtaining a distance image based on the light receiving signal of the infrared light sensor. The sensor device also includes a calibration processing unit (calibration unit F1) that performs a calibration process to identify identical points appearing in both the distance image obtained by the sensor device and the captured image obtained by the tracking camera. The calibration process identifies the positions of identical parts between a sensor device simple model, which is a simple model of the subject generated from the distance image obtained by the sensor device, and a camera simple model, which is a simple model of the subject generated from the captured image obtained by the tracking camera. By performing the above-described calibration process, it is possible to appropriately convert a target tracking image frame defined in a three-dimensional coordinate system by the sensor device into an image frame in the tracking camera coordinate system. Furthermore, since the calibration method uses a simple model of the subject, calibration can be achieved based on detected parts of a simple model of the subject used for calibration, such as a subject with both hands raised.

[0262] Furthermore, in an information processing device according to an embodiment, the number of sensor devices and the number of tracking cameras are multiple, and the calibration processing unit performs a unit sampling process for each sensor device while switching between target sensor devices. The unit sampling process samples the positions of each part of the sensor device-side simple model and the camera-side simple model, with the composition adjusted so that each tracking camera captures the ranging range of the target sensor device. By performing the unit sampling process for each sensor device, sample data necessary for calibration (identification of identical points) between each tracking camera and each sensor device can be acquired. In each unit sampling process, the imaging range of each tracking camera can be sufficiently overlapped with the ranging range of the target sensor device, so that calibration between each tracking camera and the target sensor device can be directly performed. Therefore, by performing the unit sampling process for each sensor device while switching between target sensor devices as described above, calibration between each tracking camera and each sensor device can be performed with high accuracy.

[0263] In addition, in the information processing device as an embodiment, there are multiple sensor devices and multiple tracking cameras, and the tracking cameras include a first tracking camera that is positioned so that it can capture the imaging ranges of all other tracking cameras and the ranging ranges of all sensor devices, and a second tracking camera that is positioned on either the left or right side of the first tracking camera and is positioned so that it can capture the imaging range of the first tracking camera, and the sensor devices include a same-side sensor device and an opposite-side sensor device that are positioned on the same side as the second tracking camera and on the opposite side as the first tracking camera, and the calibration processing unit performs calibration processing between the first tracking camera and the same-side sensor device based on the identification result of the same points between the second tracking camera and the first tracking camera and the result of calibration processing between the first tracking camera and the same-side sensor device. If calibration processing between each tracking camera and each sensor device is to be performed with a single sampling process, the imaging range of each tracking camera needs to be sufficiently overlapped with the ranging ranges of all sensor devices, but for the second tracking camera, if its imaging range is to be sufficiently overlapped with the ranging range of the opposite-side sensor device, it may not be possible to sufficiently capture the ranging range of the same-side sensor device, in which case calibration processing between the second tracking camera and the same-side sensor device cannot be performed directly.With the above configuration, even if calibration processing between the second tracking camera and the same-side sensor device cannot be performed directly, calibration processing can be performed along with the first tracking camera. In other words, when realizing the calibration processes between the second tracking camera and the opposite-side sensor device, and between the second tracking camera and the same-side sensor device, there is no need to perform separate sampling processes for calibration in a state where the imaging range of the second tracking camera is sufficiently overlapped with the ranging range of the opposite-side sensor device, and in a state where the imaging range of the second tracking camera is sufficiently overlapped with the ranging range of the same-side sensor device, and the calibration processes between each tracking camera and each sensor device can be realized with a single sampling process.

[0264] Furthermore, in the information processing device according to the embodiment, the event to be imaged by the tracking camera is a live music event, so that the tracking target subject can be appropriately specified by the tracking camera when the event to be imaged is a live music event.

[0265] In addition, an information processing method as an embodiment is an information processing method in which an information processing device performs processing to allow a user to specify a tracking target subject by a tracking camera capable of tracking a subject from subjects appearing in a sensing image obtained by a sensor device having a light receiving sensor separate from the tracking camera. With such an information processing method, it is possible to obtain the same functions and effects as the information processing device as the above-described embodiment.

[0266] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0267] 9. The Present Technology The present technology may also have the following configuration: (1) An information processing device comprising: a designation reception processing unit that performs processing to have a user designate a tracking target subject by a tracking camera capable of tracking a subject from subjects captured in a sensed image captured by a sensor device having a light receiving sensor separate from the tracking camera. (2) The sensor device has, as the light receiving sensors, a visible light sensor that can receive visible light and an infrared light sensor that can receive infrared light, and has an imaging function that obtains an image captured for visible light based on a light receiving signal of the visible light sensor and a distance measuring function that obtains a distance image based on the light receiving signal of the infrared light sensor, and the designation reception processing unit performs processing to display an image based on the distance image as an image for accepting designation of the tracking target subject when brightness of the captured image becomes equal to or lower than a predetermined brightness in a state where the image captured by the sensor device is displayed as an image for accepting designation of the tracking target subject. (3) The information processing device according to (1) or (2), wherein the designation reception processing unit, in response to the tracking target subject being designated from a subject appearing in a sensing image captured by the sensor device, performs a process of displaying name information of the tracking target subject at a position corresponding to a detection frame of the tracking target subject superimposed on a display image of the sensing image. (4) The information processing device according to any of (1) to (3), wherein the designation reception processing unit, in response to the tracking target subject being designated from a subject appearing in the sensing image, displays an image of the designated tracking target subject extracted from the sensing image in a tracking target display area provided on the operation screen that displays name information of the tracking target subject. (5) The information processing device according to any one of (1) to (4), wherein the specification reception processing unit receives a specification of a tracking composition of the subject by the tracking camera as a specification of a tracking image frame for a display image of a virtual simple model that is a virtual simple model of the subject.(6) The information processing device according to (5), wherein the specification reception processing unit receives a specification of a tracking composition of the subject by the tracking camera as a specification of a position of the tracking image frame relative to a display image of the virtual simple model and a size of the tracking image frame. (7) The information processing device according to (6), wherein the sensor device has an infrared light sensor capable of receiving infrared light and has a distance measurement function for obtaining a distance image based on a light receiving signal of the infrared light sensor, and further comprises an image frame calculation unit for calculating a target image frame of the tracking camera for tracking the tracking subject based on three-dimensional position information of a detection simple model which is a simple model of the tracking subject generated based on the distance image obtained by the sensor device, wherein the specification reception processing unit performs a process for storing information indicating a positional deviation of the image frame center of the tracking image frame relative to the virtual simple model as relative model image frame center positional deviation information, and the image frame calculation unit calculates the target image frame so that the positional deviation indicated by the relative model image frame center positional deviation information is reproduced as a positional deviation of the image frame center of the target image frame relative to the detection simple model. (8) The information processing device described in (7) above, wherein the specification reception processing unit calculates the vertical positional deviation amount, which indicates the vertical positional deviation amount included in the relative model image frame center positional deviation information, as a relative value scaled by a first scaler that uses the vertical position of a specified part of the virtual simple model as a reference value and the length between the specified parts spaced apart in the vertical direction in the virtual simple model as a reference unit, and the image frame calculation unit determines the vertical center position of the target image frame to the vertical position specified by the vertical positional deviation amount in a second scaler that uses the vertical position of the specified part of the detected simple model as a reference value and the length between the specified parts spaced apart in the vertical direction in the detected simple model as a reference unit.(9) The sensor device has as the light receiving sensor a visible light sensor capable of receiving visible light and an infrared light sensor capable of receiving infrared light, and has an imaging function of obtaining an image of visible light based on a light receiving signal of the visible light sensor, and a distance measuring function of obtaining a distance image based on the light receiving signal of the infrared light sensor, and is provided with an image frame calculation unit that calculates a target image frame of the tracking camera for tracking the tracking target subject based on three-dimensional position information of a detection simple model that is a simple model of the tracking target subject generated based on the distance image obtained by the sensor device, and the specification reception processing unit performs processing to display a specification reception screen on which a plurality of virtual simple models that are virtual simple models of the subjects are arranged as a specification reception screen for the tracking target subject during group tracking of a plurality of subjects as tracking target subjects of the tracking camera, and receives specification of parts from each of the arranged virtual simple models, and the image frame calculation unit The information processing device according to any one of (1) to (8), wherein, when the simple detection model of each subject set as a tracking target of the group tracking is obtained, the target image frame is calculated so that a part specified on the specification receiving screen out of detected parts of each subject set as the tracking target is included within the target image frame. (10) The information processing device according to (1), wherein the tracking camera is configured to be able to perform switching control of the tracking sensitivity of the tracking camera while the tracking camera is tracking the tracking target subject. (11) The information processing device according to any one of (1) to (10) is configured to be able to execute: designated tracking composition control that controls the composition of the tracking camera so that the tracking composition is the tracking composition designated by a user for the subject to be tracked; and semi-automatic composition control that selects a composition of the tracking camera based on a composition selection table in which weight information is associated with each combination of the subject to be imaged by the tracking camera and the composition type of the tracking camera, and controls the composition of the tracking camera so that the selected composition is achieved, wherein the designation reception processing unit receives an instruction operation to execute the designated tracking composition control and an instruction operation to execute the semi-automatic composition control.(12) The information processing device according to any one of (1) to (11), wherein the sensor device has an infrared light sensor capable of receiving infrared light as the light receiving sensor and has a distance measurement function of obtaining a distance image based on a light reception signal of the infrared light sensor, and comprises a calibration processing unit that performs a calibration process to identify identical points that appear in both the distance image obtained by the sensor device and the captured image obtained by the tracking camera, by identifying positions of identical parts between a sensor device-side simple model that is a simple model of the subject generated from the distance image obtained by the sensor device and a camera-side simple model that is a simple model of the subject generated from the captured image obtained by the tracking camera. (13) The information processing device according to (12), wherein there are a plurality of sensor devices and a plurality of tracking cameras, and the calibration processing unit performs a unit sampling process for each sensor device while switching between the target sensor devices, sampling positions of parts of the sensor device-side simple model and the camera-side simple model with the composition adjusted so that each tracking camera captures an image within the distance measurement range of one target sensor device. (14) The information processing device according to (12), wherein the number of the sensor devices and the tracking cameras is plural, the tracking cameras include a first tracking camera arranged so as to be able to capture the imaging ranges of all the other tracking cameras and the ranging ranges of all the sensor devices, and a second tracking camera arranged on either the left or right side as viewed from the first tracking camera and arranged so as to be able to capture the imaging range of the first tracking camera, the sensor devices include a same-side sensor device and an opposite-side sensor device arranged at a position on the same side as the second tracking camera as viewed from the first tracking camera and a position on the opposite side as viewed from the second tracking camera, and the calibration processing unit performs the calibration process between the first tracking camera and the same-side sensor device based on the identification result of the same points between the second tracking camera and the first tracking camera and the result of the calibration process between the first tracking camera and the same-side sensor device.(15) The information processing device according to any one of (1) to (15), wherein the event to be imaged by the tracking camera is a live music event. (16) An information processing method, in which the information processing device performs a process of having a user designate a tracking target subject to be tracked by a tracking camera capable of tracking a subject from subjects appearing in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera. (17) A recording medium having a program readable by a computer device recorded thereon, the recording medium recording the program causing the computer device to realize a designation acceptance processing function of performing a process of having a user designate a tracking target subject to be tracked by a tracking camera capable of tracking a subject from subjects appearing in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera.

[0268] 100 Image processing system 1 Information processing device 2 Parent camera 3, 3-1, 3-2, 3-3 Tracking camera 4, 4-1, 4-2, 4-3 Pan head 5 Switcher 6, 6-1, 6-2, 6-3, 6-4 Distance measurement / imaging device 11 CPU 19 Storage unit 20 Communication unit Gm Operation screen Gs Setting screen Arc Camera name display area Are Production area Art Target area At Tracking target display area t1, t2, t3, t4, t5, t6 Individual display area ti Image display area tn Name display area Bm Execution instruction button B0 Menu button B1 Semi-auto ON / OFF button It Tracking indicator Acn Controller name display area Bpd Pull-down button B2 Common setting button B3 Control setting button Bs Start button As1 Camera switching operation area As2 Subject name setting area As3 Controller setting area bcm Camera name input box bcn Controller name input box bt1, bt2, bt3, bt4, bt5, bt6 Subject name input box Bp1, Bp2 Add button Be1, Be2 Delete button Bc Camera switching button As4 Production setting area As5 Preset setting area cb, cb1, cb2, cb3, cb4, cb5, cbs, cbt Check box bpn Preset name input box Fd Detection frame Ims Captured image Mbv Virtual simple model Fs Image frame Vc Vertical center line J Intersection Vrc Vertical position deviation amount Hof Horizontal offset amount Mbd Detection simple model cba, cbb Check box B9 OK button B10 Cancel button F1 Calibration section F2 Designation reception section F3 Operation reception section F4 Composition selection section F5 Composition switching control section F6 Weight update section F7 Image recognition processing section F8 Image frame calculation section F9 Coordinate calculation section F10 Platform and camera control section F11 Cutout image generation section cbg Check box 30 Joystick

Claims

1. An information processing device having a designation reception processing unit that allows a user to designate a subject to be tracked by a tracking camera capable of tracking the subject from among subjects appearing in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera.

2. The information processing device according to claim 1, wherein the sensor device has as its light receiving sensors a visible light sensor capable of receiving visible light and an infrared light sensor capable of receiving infrared light, an imaging function for obtaining an image of visible light based on the light receiving signal of the visible light sensor, and a distance measurement function for obtaining a distance image based on the light receiving signal of the infrared light sensor, and wherein the designation reception processing unit, when an image captured by the sensor device is displayed as an image for accepting designation of the subject to be tracked, performs processing to display an image based on the distance image as an image for accepting designation of the subject to be tracked in response to the brightness of the image becoming equal to or lower than a predetermined brightness.

3. The information processing device according to claim 1, wherein the designation reception processing unit performs processing to display name information of the tracking target subject at a position corresponding to the detection frame of the tracking target subject superimposed on the display image of the sensing image in response to the tracking target subject being designated from a subject appearing in the sensing image captured by the sensor device.

4. The information processing device according to claim 1, wherein the designation reception processing section displays a sensing image captured by the sensor device on an operation screen on which it is possible to perform an instruction operation as to whether or not to track the tracking target subject using at least the tracking camera, and, in response to the tracking target subject being designated from a subject appearing in the sensing image, displays an image of the designated tracking target subject extracted from the sensing image in a tracking target display area on the operation screen that displays name information of the tracking target subject.

5. The information processing device according to claim 1, wherein the specification reception processing unit receives a specification of a tracking composition for the subject by the tracking camera as a specification of a tracking image frame for a display image of a virtual simple model that is a virtual simple model of the subject.

6. The information processing device according to claim 5, wherein the specification reception processing unit receives a specification of a tracking composition of the subject by the tracking camera as a specification of a position of the tracking image frame relative to a display image of the virtual simple model and a size of the tracking image frame.

7. The information processing device according to claim 6, wherein the sensor device has an infrared light sensor capable of receiving infrared light as the light receiving sensor, and has a distance measurement function for obtaining a distance image based on the light receiving signal of the infrared light sensor, and is equipped with an image frame calculation unit that calculates a target image frame of the tracking camera for tracking the tracking subject based on three-dimensional position information of a detection simple model that is a simple model of the tracking subject generated based on the distance image obtained by the sensor device, and the specification reception processing unit performs a process of storing information indicating a positional deviation of the image frame center of the tracking image frame relative to the virtual simple model as relative model image frame center positional deviation information, and the image frame calculation unit calculates the target image frame so that the positional deviation indicated by the relative model image frame center positional deviation information is reproduced as a positional deviation of the image frame center of the target image frame relative to the detection simple model.

8. The information processing device according to claim 7, wherein the specification reception processing unit calculates the vertical positional deviation amount, which indicates the vertical positional deviation amount included in the relative model image frame center positional deviation information, as a relative value scaled by a first scaler that uses the vertical position of a specified part of the virtual simple model as a reference value and the length between the specified parts spaced apart in the vertical direction in the virtual simple model as a reference unit, and the image frame calculation unit determines the vertical center position of the target image frame to the vertical position specified by the vertical positional deviation amount using a second scaler that uses the vertical position of the specified part of the detected simple model as a reference value and the length between the specified parts spaced apart in the vertical direction in the detected simple model as a reference unit.

9. The sensor device has as the light receiving sensors a visible light sensor capable of receiving visible light and an infrared light sensor capable of receiving infrared light, and has an imaging function for obtaining an image of visible light based on the light receiving signal of the visible light sensor, and a distance measuring function for obtaining a distance image based on the light receiving signal of the infrared light sensor, and is provided with an image frame calculation unit that calculates a target image frame of the tracking camera for tracking the tracking target subject based on three-dimensional position information of a detection simple model that is a simple model of the tracking target subject generated based on the distance image obtained by the sensor device, and the specification reception processing unit performs processing for displaying a specification reception screen on which a plurality of virtual simple models that are virtual simple models of the subjects are arranged as a specification reception screen for the tracking target subject during group tracking of a plurality of subjects as tracking target subjects of the tracking camera, and receives specification of parts from each of the plurality of arranged virtual simple models, and the image frame calculation unit 2. The information processing device according to claim 1, wherein, when the simple detection model of each subject that is set as a tracking target of the group tracking has been obtained, the target image frame is calculated so that a part specified on the specification acceptance screen out of the detected parts of each subject that is set as the tracking target is included within the target image frame.

10. The information processing device according to claim 1, wherein the tracking sensitivity of the tracking camera can be switched while the tracking camera is tracking the tracking target subject.

11. An information processing device as described in claim 1, configured to be able to execute: specified tracking composition control that controls the composition of the tracking camera so that the tracking composition is the tracking composition specified by the user for the subject to be tracked; and semi-automatic composition control that selects a composition for the tracking camera based on a composition selection table in which weighting information is associated with each combination of the subject to be imaged by the tracking camera and the composition type of the tracking camera, and controls the composition of the tracking camera so that the selected composition is achieved, wherein the specification reception processing unit receives an instruction operation to execute the specified tracking composition control and an instruction operation to execute the semi-automatic composition control.

12. The information processing device according to claim 1, wherein the sensor device has an infrared light sensor capable of receiving infrared light as the light receiving sensor, and has a distance measurement function for obtaining a distance image based on the light receiving signal of the infrared light sensor, and is provided with a calibration processing unit that performs a calibration process for identifying the same points that appear in both the distance image obtained by the sensor device and the captured image obtained by the tracking camera, by identifying the positions of the same parts between a sensor device side simple model, which is a simple model of the subject generated from the distance image obtained by the sensor device, and a camera side simple model, which is a simple model of the subject generated from the captured image obtained by the tracking camera.

13. An information processing device as described in claim 12, wherein the number of the sensor devices and the number of the tracking cameras are each plural, and the calibration processing unit executes a unit sampling process for each sensor device while switching the target sensor device, sampling the positions of parts of the sensor device side simple model and the camera side simple model with the composition adjusted so that each tracking camera captures the ranging range of one of the target sensor devices.

14. The information processing device according to claim 12, wherein there are a plurality of the sensor devices and a plurality of the tracking cameras, and the tracking cameras include a first tracking camera arranged so as to be able to capture the imaging ranges of all the other tracking cameras and the ranging ranges of all the sensor devices, and a second tracking camera arranged on either the left or right side of the first tracking camera and arranged so as to be able to capture the imaging range of the first tracking camera, and the sensor devices include a same-side sensor device and an opposite-side sensor device arranged at a position on the same side as the second tracking camera and a position on the opposite side as the second tracking camera, and the calibration processing unit performs the calibration process between the first tracking camera and the same-side sensor device based on the result of identifying the same point between the second tracking camera and the first tracking camera and the result of the calibration process between the first tracking camera and the same-side sensor device.

15. The information processing device according to claim 1, wherein the event to be imaged by the tracking camera is a live music event.

16. An information processing method in which an information processing device performs processing to allow a user to specify a subject to be tracked by a tracking camera capable of tracking a subject from among subjects appearing in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera.

17. A recording medium having a program readable by a computer device recorded thereon, which causes the computer device to realize a designation reception processing function that allows a user to designate a subject to be tracked by a tracking camera capable of tracking a subject from among subjects captured in a sensing image captured by a sensor device having a light receiving sensor separate from the tracking camera.