Program, information processing device, and information processing method

Abstract

This information processing device executes transmission image change control which, when an image of a first area in a captured image is used as a transmission image and in response to detection of a detection target associated with a second area in the second area, which is set to have a narrower viewing angle range than the first area, uses an image of the second area in the captured image as a transmission image.

Description

Program, information processing device, and information processing method

[0001] The present technology relates to a program, an information processing device, and an information processing method, and to a technology for switching images to be transmitted.

[0002] In recent years, it has become easy for general users to create and distribute video content using terminal devices such as cameras and smartphones.

[0003] Furthermore, Patent Documents 1, 2 and 3 listed below disclose techniques for using the detection results in a partial area (box) within an image for specific processing.

[0004] Japanese Patent Application Laid-Open No. 2006-350979 Japanese Patent Application Laid-Open No. 2021-128797 Special Publication No. 2010-536087

[0005] For example, when a user places a camera or smartphone in a fixed position and films themselves as a performer, they may want to make themselves or an object appear larger or take a wide-angle shot. To do this, they must move themselves or the object closer or further away from the camera. This is because it is difficult for users to change the camera's angle of view or instruct cropping within a specific range during a performance. This makes it difficult for users to create the video content they desire.

[0006] Therefore, the present disclosure proposes a technology that enables a user to easily switch the angle of view in video content as desired, for example.

[0007] A program according to the present technology causes an information processing device to execute a send-image change control in which, when an image of a first area in a captured image is set as a send image, an image of the second area in the captured image is set as a send image in response to detection that a detection target associated with a second area, the second area being set as a narrower angle of view range than the first area, is included in the captured image. The information processing device according to the present technology executes the send-image change control based on such a program. For example, a second area having a narrower range than the first area in the captured image is set and associated with a subject part. The detection target is, for example, a part of a human body that is set as a detection target, such as a hand, wrist, face, or foot, but is not limited to a human part and may also be a specific object. When a captured image of a specific detection target falls within the second area in the captured image, the image of the second area is set as a send image.

[0008] 1 is an explanatory diagram of a system configuration according to an embodiment of the present technology. FIG. 1 is an explanatory diagram of how video content is shot. FIG. 1 is an explanatory diagram of how video content is shot. FIG. 1 is an explanatory diagram of how video content is shot. FIG. 1 is a block diagram of an information processing device applicable to a terminal device or the like according to an embodiment. FIG. 2 is an explanatory diagram of a setting screen according to an embodiment. FIG. 3 is a flowchart of setting processing according to an embodiment. FIG. 4 is a flowchart of frame setting processing according to an embodiment. FIG. 5 is an explanatory diagram of preventing frame overlap according to an embodiment. FIG. 6 is an explanatory diagram of preventing frame overlap according to an embodiment. FIG. 7 is a flowchart of another example of frame setting processing according to an embodiment. FIG. 8 is an explanatory diagram of a monitor screen according to an embodiment. FIG. 9 is an explanatory diagram of a monitor screen according to an embodiment. FIG. 10 is an explanatory diagram of image switching using a second area frame according to an embodiment. FIG. 11 is an explanatory diagram of image switching with a return frame added according to an embodiment. FIG. 12 is an explanatory diagram of image switching with a detection frame added according to an embodiment. FIG. 13 is an explanatory diagram of arrangements of detection frames and return frames according to an embodiment. FIG. 14 is an explanatory diagram of image switching using a zoom operation according to an embodiment. FIG. 15 is an explanatory diagram of image switching using multiple cameras according to an embodiment. FIG. 16 is an explanatory diagram of skeleton point display according to an embodiment. FIG. 17 is an explanatory diagram of skeleton point display according to an embodiment. FIG. 18 is an explanatory diagram of correspondence between skeleton points and second area frames according to an embodiment. FIG. 19 is an explanatory diagram of a case where multiple conditions are satisfied according to an embodiment. 10 is an explanatory diagram of an example of cropping processing by a camera according to a first embodiment. FIG. 10 is a block diagram of the functional configuration of the first embodiment. FIG. 10 is an explanatory diagram of an example of cropping processing by a terminal device according to a second embodiment. FIG. 10 is a block diagram of the functional configuration of the second embodiment. FIG. 10 is an explanatory diagram of an example of switching between multiple cameras according to a third embodiment. FIG. 10 is a block diagram of the functional configuration of the third embodiment. FIG. 10 is an explanatory diagram of an example of switching between multiple cameras according to a fourth embodiment. FIG. 10 is a block diagram of the functional configuration of the fourth embodiment. FIG. 10 is an explanatory diagram of an example of switching using a PTZ camera according to a fifth embodiment. FIG. 10 is a block diagram of the functional configuration of the fifth embodiment. FIG. 10 is a flowchart of monitor processing according to an embodiment. FIG. 10 is a flowchart of monitor screen control according to an embodiment. FIG. 10 is a flowchart of monitor screen control according to an embodiment. FIG. 10 is a flowchart of condition determination for a second area frame according to an embodiment. FIG. 10 is a flowchart of condition determination for a second area frame according to an embodiment.1 is a flowchart of determining a release condition for a second area frame according to an embodiment. FIG. 2 is a flowchart of determining a release condition for a second area frame according to an embodiment. FIG. 3 is a flowchart of controlling second area image transmission according to an embodiment. FIG. 4 is a flowchart of controlling second area image transmission according to an embodiment. FIG. 5 is a flowchart of controlling second area image transmission according to an embodiment. FIG. 6 is a flowchart of controlling second area image transmission according to an embodiment. FIG. 7 is an explanatory diagram of adjustment of a second area frame according to an embodiment. FIG. 8 is a flowchart of automatic adjustment of a second area frame according to an embodiment. FIG. 9 is a flowchart of manual adjustment of a second area frame according to an embodiment.

[0009] The embodiments will be described below in the following order: <1. System configuration and shooting mode> <2. Configuration of information processing device> <3. Setting process> <4. Image switching during shooting> <5. Configuration of each embodiment> <6. Example of transmitted image change control process> <7. Summary and modifications>

[0010] In this disclosure, the term "image" includes both moving images and still images. In the embodiments, examples of capturing moving images will be mainly described, and the term "image" will include such moving images.

[0011] 1 shows an example of a system that allows a user to create video content to be distributed. This system includes a terminal device 1, a camera 2, and a distribution server 3. The user then uses the terminal device 1 and camera 2 to shoot video to create video content and upload it to the distribution server 3.

[0012] Camera 2 represents an imaging device used by a user to create video content, and the type of camera is not particularly important. Examples include a general camera (compact digital camera, interchangeable lens camera, etc.) shown as camera 2A, and a PTZ (pan-tilt-zoom) camera 2B. PTZ camera 2B is a camera that can pan and tilt the shooting direction at the location where it is placed, and can change the angle of view by zooming. Although not shown, camera 2A may be equipped with a camera platform that allows panning and tilting.

[0013] The type of camera 2 to be used for shooting is up to the user, but in the embodiments of the present disclosure, examples will be given of using one camera 2A, using one PTZ camera 2B, using multiple cameras 2A, and using a camera 2A and a PTZ camera 2B.

[0014] The terminal device 1 is an information processing device used by a user who creates video content. Specific examples of the terminal device 1 include a smartphone 1A, a PC (personal computer) 1B, and a tablet device 1C. In this embodiment, the terminal device 1 has functions such as sending video content for uploading, controlling the camera 2, and displaying the shooting status to the user on a monitor.

[0015] The distribution server 3 is a server that distributes video content uploaded by users to the general public via the network 4. The distribution server 3 is composed of one or more information processing devices, and performs processes such as transmitting information to the terminal device 1 and accepting information uploads from the terminal device 1.

[0016] The terminal device 1 is capable of communicating with the distribution server 3 via a network 4. The network 4 may be, for example, the Internet, a home network, a LAN (Local Area Network), a mobile communication network, or any other type of network.

[0017] The terminal device 1 and the camera 2 are also capable of communicating with each other. For example, the terminal device 1 and the camera 2 can communicate with each other via short-range wireless communication or wired communication. They may also be capable of communicating with each other via a network 4. This allows the terminal device 1 to receive image data as captured images from the camera 2 and to transmit control signals to the camera 2.

[0018] With this system, the terminal device 1 can send the captured images received from the camera 2 to the distribution server 3 in real time as video content, or edit the images and send them to the distribution server 3 .

[0019] 2, 3, and 4 show examples of how user 101 takes pictures. FIG. 2 shows an example in which only one camera 2A is used. The camera 2A, which has a fixed imaging direction and imaging angle of view, is used to capture the user 101. The user 101 acts as a performer and takes a video, for example, introducing an item 6. At this time, the user 101 can monitor the shooting situation by placing a terminal device 1, for example, a smartphone 1A, nearby.

[0020] In this embodiment, when a single camera 2A is used in this manner, crop processing is used to switch between images with different angles of view or positions in the transmitted images transmitted by the terminal device 1. In this disclosure, the term "transmitted image" refers to an image that the terminal device 1 transmits and outputs to the outside, for example, as video content, and corresponds to, for example, an image that the terminal device 1 outputs as video content for uploading to the distribution server 3. In addition, even when the terminal device 1 records video content on a recording medium rather than directly uploading it, the image output as the video content corresponds to a transmitted image.

[0021] Therefore, for example, the transmitted image is an image that is provided as is as video content, but in this embodiment, it is possible to output a transmitted image that is appropriately switched between an image of a first area and an image of a second area that is set to have a narrower angle of view range than the first area in the image captured by camera 2.

[0022] The first area is, for example, the entire area of ​​the image captured by camera 2A, and is an image captured in the imaging direction and angle of view set in camera 2A. For example, it is the entire range of the image captured when user 101 normally takes a picture with camera 2A. The second area is, for example, a partial region of the captured image. For example, it is a partial area of ​​the entire image that captures the hands of user 101. For example, the image of the second area is a cropped image obtained by cropping (cutting out) a partial area of ​​the image captured by camera 2A.

[0023] This allows the transmitted image to be a moving image that is successively switched between a full image showing a wide view of the scene in which the user 101 is performing and a cropped image showing, for example, only the hands of the user 101. The transmitted image change control that switches between the image in the first area and the image in the second area is then executed by the terminal device 1 based on the intention of the user 101.

[0024] As will be described in detail later with reference to FIG. 12 , the entire range of the captured image of camera 2A is defined as the first area, and during shooting, a first area image 24 is displayed as the captured image on the screen of terminal device 1 as shown in FIG. 12 . A second area frame 27 indicating the second area is displayed within the first area image 24. A detection target 26 is also set for the second area. For example, the left wrist of user 101 is set as the detection target. In this case, when it is detected that the captured image of the left wrist has entered the second area frame 27, the terminal device 1 performs control to change the image to be sent from the full image to a cropped image. Furthermore, for example, when it is detected that the left wrist has left the second area, the terminal device 1 performs control to change the image to be sent from the cropped image to the full image.

[0025] For example, by the terminal device 1 performing such processing, the user 101 can switch between a full image and a cropped image by moving his or her body during the demonstration, and by appropriately setting the second area frame 27, the cropped image can be an image suitable for introducing, for example, an item 6.

[0026] 3 shows an example in which two cameras 2A are used for filming. In the figure, two cameras 2A are shown as cameras 2A1 and 2A2. Here, assume that a video content is being produced in which user 101 is cooking. For example, camera 2A1 is a camera that captures an overall image and is positioned so that it can capture images from directly in front of user 101 with the required angle of view. Camera 2A2 is positioned and has an angle of view set so that it can capture close-up images of the food from above.

[0027] In this case, the image captured by camera 2A1 is the image of the first area, and the image captured by camera 2A2 is the image of the second area, and the terminal device 1 (smartphone 1A, etc.) sequentially controls the switching between these images and outputs the transmitted image. Therefore, the video content will be one in which the cooking scene viewed from the front and close-ups of the cooking are sequentially switched.

[0028] FIG. 4 shows an example in which a camera 2A and a PTZ camera 2B are used for filming. For example, camera 2A is a camera that captures an overall image of a scene in which user 101 is performing, and is set to capture images from the front of user 101 with the required angle of view. The image captured by this camera 2A is an image of a first area. PTZ camera 2B is placed near camera 2A and set to capture images with a predetermined shooting direction and zoom angle of view. The image captured by this PTZ camera 2B is an image of a second area. By using PTZ camera 2B, when multiple second areas are set, only one PTZ camera 2B can be used to cover each second area. In other words, one second area can be set by setting the pan / tilt direction and zoom angle of view, but multiple second areas can be set by providing multiple settings for the pan / tilt direction and zoom angle of view.

[0029] The terminal device 1 (PC 1B, etc.) can sequentially control switching between the image in the first area and the image in one or more second areas, and output the transmitted image.

[0030] 2, 3, and 4, a smartphone 1A or a PC 1B is shown as an example of the terminal device 1, and the user 101 can monitor the shooting on the screen of the smartphone 1A or the like, but it is also possible to place a large monitor display near the camera 2, for example, and have the terminal device 1 display the monitor image on that monitor display.

[0031] 2 shows an example in which one camera 2A is used, but an example in which one PTZ camera 2B is used is also envisioned. In the case of camera 2A, the second area image 25 is assumed to be a cropped image, but when PTZ camera 2B is used, the second area image 25 can also be a zoomed image. Furthermore, by using PTZ camera 2B, one camera can be used to handle multiple second areas.

[0032] 5 illustrates an example of the configuration of an information processing device 70 that can be used as the terminal device 1. The information processing device 70 can be configured as, for example, a smartphone, a tablet device, a mobile terminal device, a general-purpose personal computer, or a dedicated workstation.

[0033] 5 executes various processes in accordance with programs stored in a nonvolatile memory unit 74, such as a read-only memory (ROM) 72 or an electrically erasable programmable read-only memory (EEP-ROM), or programs loaded from a storage unit 79 to a random access memory (RAM) 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes. The CPU 71 executes various control and calculation functions, which will be described later, using programs.

[0034] In addition, a processor other than the CPU 71 may be provided, such as a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), or an AI (Artificial Intelligence) processor.

[0035] The CPU 71, ROM 72, RAM 73, and nonvolatile memory unit 74 are interconnected via a bus 83. The bus 83 is also connected to an input / output interface 75.

[0036] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. A user operation is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0037] The input / output interface 75 is also connected, either integrally or separately, to a display unit 77 such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 78 such as a speaker.

[0038] The display unit 77 performs various displays as a user interface. The display unit 77 is configured, for example, by a display device provided in the housing of the information processing device 70 or a separate display device connected to the information processing device 70. The display unit 77 displays various images on the display screen based on instructions from the CPU 71. The display unit 77 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0039] The input / output interface 75 may be connected to a storage unit 79 configured with a solid-state memory chip, a solid-state drive (SSD), a hard disk drive (HDD), etc., or a communication unit 80 configured with a modem, etc. The storage unit 79 can be used to store various types of data. A database can also be constructed in the storage unit 79. The communication unit 80 performs communication processing via the network 4.

[0040] A drive 82 is also connected to the input / output interface 75 as needed, and a removable recording medium 81 such as a flash memory, a memory card, a magnetic disk, an optical disk, or a magneto-optical disk is appropriately attached. The drive 82 allows data files such as image files and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as needed.

[0041] In this information processing device 70, software can be installed via network communication by the communication unit 80 or via a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0042] The terminal device 1 can be configured with such an information processing device 70. The processing performed by the terminal device 1, which will be described later, can be realized by the hardware configuration of the information processing device 70 in FIG. 5 and software installed thereon. The distribution server 3 can also be configured with the information processing device 70 configured as shown in FIG. 5. Furthermore, the camera 2 may include the components of the information processing device 70. Although not shown in FIG. 1, a cloud server that provides cloud computing services to the terminal device 1 may be provided, and the processing of the terminal device 1 may be executed by the cloud server. Such a cloud server can also be configured with the information processing device 70 configured as shown in FIG. 5.

[0043] 3. Setting Process> The following describes the setting process performed by the terminal device 1 according to the embodiment. The setting process is a process executed by the terminal device 1 in order for the user 101 to set a second area and a detection target corresponding to the second area in the captured image.

[0044] In this embodiment, as will be described later with reference to FIG. 12 , a second area having a narrower angle of view is set relative to a first area, which is substantially the entire range of the captured image, and a second area frame 27 is displayed on the screen. When a captured image of a set detection target 26 (e.g., the left wrist) of the user 101's body part falls within the range of this second area frame 27, switching is performed between the first area image 24 (full image) and the second area image 25 (e.g., a cropped image). For this process, the user 101 sets the second area frame 27 indicating the second area in advance, and also sets the detection target 26 in association with the second area. The following describes this setting process.

[0045] 6 shows an example of a setting screen 90 displayed on the terminal device 1. This setting screen 90 is displayed when an application program is started by the terminal device 1 and a setting process for photographing is selected. For example, while setting up the camera 2 and performing photographing, the user 101 displays the setting screen 90 on the terminal device 1 at hand (for example, the smartphone 1A) and performs setting operations.

[0046] The setting screen 90 is provided with a monitor image display section 91, which displays an image currently being captured by the camera 2. The figure shows a state in which the user 101 himself is being photographed by the camera 2, and the captured image is being displayed on the monitor image display section 91. The second area frame 27 is displayed on this monitor image display section 91.

[0047] The setting screen 90 also includes a detection target selection section 93, a size selection section 94, an effect selection section 95, an effect direction selection section 96, and the like.

[0048] The detection target selection unit 93 allows the user 101 to set a specific part of the human body as the detection target 26. For example, the user 101 can select the right wrist, left wrist, left elbow, right elbow, nose, eye, right foot, left foot, etc. as the detection target 26. Note that the detection target 26 is not necessarily limited to a body part. A specific item may also be set as the detection target 26.

[0049] The size selection unit 94 allows the user 101 to select the size of the second area frame 27. The effect selection unit 95 allows the user 101 to select the type of effect for the second area. The effect direction selection unit 96 allows the user 101 to select the effect direction.

[0050] On this setting screen 90, for example, the user 101 selects the size of the second area frame 27 using the size selection unit 94, or performs a drag operation on the second area frame 27 to determine the position of the second area frame 27. The user 101 also selects the detection target 26 using the detection target selection unit 93. As a result, the detection target 26 and the second area frame 27 are set in association with each other.

[0051] 7 shows an example of the setting process of the terminal device 1. For example, this is an example of the process that the CPU 71 of the terminal device 1 executes based on a program for the setting process.

[0052] In step S101, the terminal device 1 controls the display of the setting screen 90 as shown in Fig. 6. Then, the terminal device 1 performs monitoring processes in steps S102, S103, and S104.

[0053] When the user 101 performs a basic setting operation on the setting screen 90, the terminal device 1 performs a frame setting process in step S110. Here, the basic setting operation refers to an operation of the size and position of the second area frame 27, and an operation of selecting the detection target 26. Specifically, for the purpose of explanation, the operation of the detection target selection unit 93, the operation of the size selection unit 94, and the drag operation and pinch operation on the second area frame 27 are referred to as the basic setting operations.

[0054] 8 shows an example of the frame setting process performed in step S110. When the process proceeds to step S110 after detecting that the user 101 has performed an operation to select the detection target 26 using the detection target selection unit 93, the terminal device 1 proceeds from step S111 to step S120 and sets the detection target 26.

[0055] If the operation of selecting the size of the second area frame 27 by the user 101 using the size selection unit 94 is detected and the process proceeds to step S110, the terminal device 1 proceeds from step S112 to step S121 to perform a process of changing the size of the second area frame 27. That is, the terminal device 1 changes the size setting of the range as the second area and also changes the size of the second area frame 27 displayed on the setting screen 90. Note that the user 101 can select the size of the second area frame 27 using the size selection unit 94, but the size may also be changed in step S121 in response to, for example, a pinch-in / pinch-out operation on the displayed second area frame 27.

[0056] If the user 101 proceeds to step S110 by performing an operation to change the position of the second area frame 27, such as a drag operation, the terminal device 1 proceeds from step S113 to step S122 and changes the display position of the second area frame 27.

[0057] In step S110 of FIG. 7, for example, basic settings for the second area frame 27 and the detection target 26 are made in response to the operation of the user 101 as described above.

[0058] When the user 101 performs a detailed setting operation on the setting screen 90, the terminal device 1 performs various setting processes in response to the operation in step S130. For example, the terminal device 1 performs setting processes such as effect setting and effect direction setting in response to the operation of the user 101.

[0059] Although an example of a UI (User Interface) screen is not shown, various settings such as personal settings and priority settings may also be made on the setting screen 90. For example, by registering the face of the user 101 as a personal setting, the body parts of the user 101 can be made valid as the detection target 26. For example, in the case of video content in which multiple performers are filmed, the terminal device 1 can be configured to recognize only the left wrist of a specific user 101 as the detection target 26 for image switching, and it is conceivable that face registration is performed to determine the individual.

[0060] Furthermore, multiple second area frames 27 can be set, and in this case, priority setting can be performed according to the operation of the user 101. For example, when the detection target 26 is detected in multiple second area frames 27, it can be set which second area is to be given priority.

[0061] Furthermore, an AND condition, an OR condition, etc. can be set for the multiple second area frames 27. For example, an AND condition can be set such that when the detection target 26 is detected in each of the multiple second area frames 27, the range of a specific second area frame 27 becomes the target of cropping, or an OR condition can be set such that when the detection target 26 is detected in any of the multiple second area frames 27, the range of a specific second area frame 27 becomes the target of cropping.

[0062] The first area may be set to a value other than the entire captured image. For example, the first area may be set to the entire captured image by default, but the user 101 may be allowed to set the first area arbitrarily within the range of the captured image.

[0063] The terminal device 1 executes various settings other than these basic settings in step S130 in response to an operation by the user 101.

[0064] In step S104, the terminal device 1 confirms the end of the setting operation. While the setting screen 90 is not closed, the terminal device 1 repeats steps S102 to S104, but the setting operation is terminated when the setting screen 90 is closed. In this case, the terminal device 1 proceeds from step S104 to step S140, and registers each setting (basic setting, detailed setting) that has been changed at that time as a new setting in a storage area inside the terminal device 1.

[0065] Through the above processing, the first area, the second area (second area frame 27), and the detection target 26 corresponding to the second area are set in preparation for shooting. In addition, additional detailed settings can also be made.

[0066] It is also possible to set multiple second areas (second area frames 27). For example, the user 101 may set two second area frames 27A and 27B as shown in FIG. 9A . For example, the detection target 26 corresponding to both second area frames 27A and 27B may be the right wrist, or the second area frame 27A may correspond to the right wrist and the second area frame 27B may correspond to the left wrist. However, if the frames of the same detection target overlap when multiple second area frames 27 are set, it may be difficult for the user 101 to crop the second area frame 27 intended.

[0067] For example, suppose that the detection target 26 in both second area frames 27A and 27B is the right wrist, and the user 101 moves second area frame 27B from the state shown in Fig. 9A to an overlapping state as shown in Fig. 9B. In such a case, when the right wrist enters the overlapping portion, it may be impossible to determine which range of second area frame 27A or 27B should be used as the transmitted image, and an image of a range not intended by the user 101 may be output. Therefore, a warning message 92 is displayed as shown in Fig. 9C, and when the user 101 performs an OK operation, the state before the operation shown in Fig. 9A is restored.

[0068] Furthermore, if second area frame 27A corresponds to the right wrist and second area frame 27B corresponds to the left wrist, the layout may be overlapping as shown in Fig. 10A, for example. However, suppose that user 101 then changes detection target 26 of second area frame 27B to the right wrist, as shown in Fig. 10B. This causes the same inconvenience as in Fig. 9B. Therefore, a warning message 92 is displayed as shown in Fig. 10C, and when user 101 presses OK, the state before the operation, as shown in Fig. 10A, is restored.

[0069] An example in which such a warning process is added to step S110 in Fig. 7 is shown in Fig. 11. In the following explanation, the same steps as those in the flowcharts already described will be given the same step numbers, and detailed overlapping explanations will be avoided.

[0070] In the example of FIG. 11, the basic setting operation that proceeds to step S110 in FIG. 7 also includes the operation of adding the second area frame 27.

[0071] If the process proceeds to step S110 due to the user 101's operation to add the second area frame 27, the terminal device 1 proceeds from step S114 to step S123 in Fig. 11 to perform processing to additionally display the second area frame 27 on the monitor image display unit 91. For example, the terminal device 1 displays the second area frame 27 of a default size at a default position. The terminal device 1 then proceeds to step S124.

[0072] If the process proceeds to step S110 due to the user 101 selecting the detection target 26 for a certain second area frame 27, the terminal device 1 proceeds from step S111 to step S120, where it sets the detection target 26 for the certain second area frame 27, and in this case also proceeds to step S124. Also, if the process proceeds to step S110 due to the user 101 selecting the size of a certain second area frame 27, the terminal device 1 proceeds from step S112 to step S121, where it changes the size of the corresponding second area frame 27, and then proceeds to step S124. Also, if the process proceeds to step S110 due to the user 101 changing the position of a certain second area frame 27, the terminal device 1 proceeds from step S113 to step S122, where it changes the display position of the corresponding second area frame 27, and then proceeds to step S124.

[0073] As described above, the process proceeds to step S124 when there is a possibility of overlapping of multiple second area frames 27. If no overlapping occurs, the process from step S124 to Fig. 11 ends, and the process proceeds to step S104 in Fig. 7.

[0074] On the other hand, overlapping of the second area frames 27 may occur due to the additional display of a second area frame 27 or the resizing or movement of a certain second area frame 27. However, even if overlapping occurs, there are cases where it is acceptable as shown in FIG. 10A and cases where it is not acceptable as shown in FIG. 9B. For example, if the detection targets 26 for each second area frame 27 are different, overlapping is acceptable, but if the detection targets 26 are the same, overlapping is not acceptable. Therefore, even if overlapping occurs, if it is acceptable, the processing of FIG. 11 ends from step S124.

[0075] On the other hand, if additional display of a second area frame 27 or resizing or moving of a certain second area frame 27 causes overlap of the second area frame 27 and the overlap is unacceptable, the terminal device 1 proceeds to step S125. Also, if an overlap that was acceptable up until then has occurred, but the detection target 26 for a certain second area frame 27 has been changed in step S120, causing the overlap to become unacceptable, the terminal device 1 also proceeds to step S125.

[0076] In step S125, the terminal device 1 performs control to display the warning message 92 as shown in Fig. 9C and Fig. 10C, and then waits for an OK operation on the warning message 92 in step S126.

[0077] When the OK operation by the user 101 is detected, the terminal device 1 proceeds to step S127 and returns to the state before the change that was just made. For example, the state is returned to the state before the processing of step S120, step S121, or step S122 was performed. This prevents unacceptable overlap from occurring.

[0078] When the second area frame 27 is added in step S123, the position of the added second area frame 27 may be moved rather than returning it to the state before the addition. Alternatively, when the second area frame 27 is added in step S123, it may be added and displayed at a position where it does not overlap with an existing second area frame 27.

[0079] An example of setting a return frame 28 and a detection frame 29 in addition to the second area frame 27 will be described later, but such return frame 28 and detection frame 29 are also set by the operation of the user 101 in the same way as the second area frame 27. If an unacceptable overlap occurs in these frames as well, it is advisable to perform a similar warning process.

[0080] 4. Image Switching During Shooting The following describes the processing during shooting that is performed after the above setting processing, particularly the processing for controlling the change of the transmitted image. Figures 12A and 12B show the monitor screen 15 displayed by the terminal device 1. During shooting, the user 101 can view the monitor screen 15 displayed on the terminal device 1 while acting as a performer. Note that the terminal device 1 may display the monitor screen 15 on an external display device rather than on the display unit of the terminal device 1 itself.

[0081] The monitor screen 15 is displayed when an application program is started by the terminal device 1 and processing at the time of shooting is selected. The monitor screen 15 is provided with a transmitted image display section 20, an entire image display section 21, a skeleton display button 22, and a detection execution button 23.

[0082] The detection execution button 23 is a button for selecting whether to enable or disable the image switching function using the second area frame 27 and the detection target 26. The transmission image change control described below is a process that is executed when the detection execution button 23 is on. In other words, when the detection execution button 23 is on, automatic switching of the transmission image is executed, while when the detection execution button 23 is off, the transmission image has a fixed angle of view. This detection execution button 23 allows the user 101 to select between automatic angle of view switching and a fixed angle of view, even during live distribution, for example. Note that the detection execution button 23 may not be provided, and transmission image change control may always be performed when an application program is launched, for example.

[0083] The transmitted image display section 20 displays the transmitted image transmitted from the terminal device 1. In other words, it is an image as a distributed video content, and the user 101 can thereby check the transmitted image while watching the demonstration.

[0084] The entire image display section 21 displays the entire image (first area image 24) captured by the camera 2. That is, it is an image of the entire captured range. This allows the user 101 to check the subject in the entire captured range. The first area image 24 in this entire image display section 21 displays the second area frame 27 set in the setting process described above.

[0085] Fig. 12A shows a state in which the first area image 24 (whole image) is set as the sent image. Therefore, the same captured image is displayed in the sent image display section 20 and the whole image display section 21. Fig. 12B shows a state in which the second area image 25 (e.g., a cropped image) is set as the sent image. Therefore, a portion of the captured image in the whole image display section 21 is cut out and enlarged and displayed in the sent image display section 20. Even in this case, the first area image 24 (whole image) is displayed in the whole image display section 21.

[0086] Here, it is assumed that the left wrist of the user 101 is set as the detection target 26 with respect to the second area frame 27. The terminal device 1 performs a skeleton estimation process on the user 101, who is the subject, and estimates the position of each part of the user 101's body. In the state of FIG. 12A , the left wrist of the user 101 (detection target 26) is located outside the second area frame 27. In this case, the terminal device 1 sets the first area image 24 as the sent image. FIG. 12B shows a state in which the left wrist of the user 101 is inside the second area frame 27. By detecting this state, the terminal device 1 performs control to switch the sent image to the second area image 25.

[0087] Therefore, the user 101 can freely switch between the entire image and a partially enlarged image (for example, an enlarged image obtained by cropping) of the transmitted image that is the video content by moving his / her left wrist during the demonstration.

[0088] The skeleton display button 22 is a button for selecting whether or not to display skeleton points 32 as shown in Fig. 13B , superimposed on the figure of the user 101 in the first area image 24 in the entire image display section 21. Fig. 13A shows a state in which off is selected with the skeleton display button 22, and the skeleton points 32 are not displayed superimposed on the user 101 in the entire image display section 21. Fig. 13B shows a state in which on is selected with the skeleton display button 22, and the skeleton points 32 are displayed superimposed on the user 101 in the entire image display section 21.

[0089] Displaying the skeleton points 32 makes it easier for the user 101 to recognize the position of his or her own body parts in the whole image display section 21. In other words, it becomes easier to recognize the positional relationship between the body part set as the detection target 26 and the second area frame 27. This makes it easier for the user 101 to more accurately move the body part set as the detection target 26 in and out of the second area frame 27 (an image switching operation for the user 101).

[0090] Below, various examples of image switching operations by the user 101, that is, various examples of image switching determinations made by the terminal device 1 in response to recognition of the detection target 26, will be described. Note that each of Figures 14A to 19D shows the second area frame 27 and the range of the image captured by the camera 2 that will become the transmitted image. The dotted range is the range that will become the transmitted image. Also, it is assumed that the hand (right hand or left hand) of the user 101 is the detection target 26.

[0091] FIG. 14A shows a first area image 24 which is an image of the entire range of the image captured by the camera 2, and a second area frame 27 which is set within a partial range within the first area image 24.

[0092] 14A shows a state in which the detection target 26 is not within the second area frame 27, and in this case, the terminal device 1 performs control to set the first area image 24 as the image to be sent. Assume that the terminal device 1 detects that the detection target 26 has entered the second area frame 27 on the captured image as shown in FIG. 14B due to the behavior of the user 101 at a certain point in time. At this time, the terminal device 1 sets the second area image 25, which is an image within the range of the second area frame 27, as the image to be sent, as shown in FIG. 14C. For example, the second area image 25 is sent by cropping the range of the second area frame 27 in the captured image.

[0093] Thereafter, it is assumed that the terminal device 1 detects that the detection target 26 has moved out of the second area frame 27 in the captured image as shown in Fig. 14D due to the behavior of the user 101 at a certain point in time. At this time, the terminal device 1 performs control to return the transmitted image from the state shown in Fig. 14D to the first area image 24 as shown in Fig. 14A.

[0094] As described above, the terminal device 1 controls switching of the transmitted image between the first area image 24 (e.g., a full image) and the second area image 25 (e.g., a cropped image) depending on whether the detection target 26 has entered or exited the second area frame 27. Note that the detection of whether the detection target 26 has entered or exited the second area frame 27 may be detected by detecting whether the detection target 26 has entered or exited a frame set based on the second area frame 27, without using the second area frame 27 itself, which is the target range for cropping or the like. For example, the second area frame 27 may be narrowed by a certain ratio, and a frame inside the second area frame 27 may be used for detection, and the detection target 26 entering or exiting the detection frame may be determined as the detection target 26 having entered or exited the second area frame 27. Furthermore, a slightly wider frame may be set around the second area frame 27 for detection, and the detection target 26 entering or exiting the frame may be determined as the detection target 26 entering or exiting the second area frame 27. That is, detection of the detection target 26 being included in the second area frame 27 does not require a strict determination of the second area frame 27 itself, but rather requires that it can be roughly evaluated as having entered the second area frame 27. Similarly, detection of the detection target 26 no longer being included in the second area frame 27 requires that it can be roughly evaluated as having left the second area frame 27.

[0095] 14B , when detecting that the detection target 26 has entered the second area frame 27, it is preferable to determine that the detection target 26 has entered the second area frame 27 not by detecting a momentary entry, but by continuously remaining in the second area frame 27 for a certain period of time (e.g., one to two seconds or more). Similarly, when detecting that the detection target 26 has left the second area frame 27, as shown in FIG. 14D , it is preferable to detect that the detection target 26 has been outside the second area frame 27 for a certain period of time. This is to prevent the detection target 26 from frequently entering and leaving the second area frame 27 due to the user 101's behavior, causing the transmitted image to be rapidly switched between the full image and a cropped image unintentionally by the user 101. This type of processing may also be performed in the following examples to determine whether the detection target 26 has entered or exited various frames.

[0096] Next, an example will be described in which a separate return frame 28 is provided to switch from the second area image 25 to the first area image 24. Fig. 15A shows an example in which the return frame 28 is set. Fig. 15A shows a state in which the detection target 26 is not within the second area frame 27, and the terminal device 1 sets the first area image 24 as the sent image. When the terminal device 1 detects that the detection target 26 has entered the second area frame 27 on the captured image due to the behavior of the user 101 at a certain point in time, as shown in Fig. 15B, the terminal device 1 sets the second area image 25 (e.g., a cropped image) as the sent image, as shown in Fig. 15C.

[0097] Thereafter, it is assumed that the terminal device 1 detects that the detection target 26 has moved out of the second area frame 27 on the captured image as shown in Fig. 15D due to the behavior of the user 101. However, the terminal device 1 maintains the sent image as the second area image 25. On the other hand, when it detects that the detection target 26 has entered the return frame 28 on the captured image as shown in Fig. 15E due to the behavior of the user 101, the terminal device 1 performs control to return the sent image from the state shown in Fig. 15E to the first area image 24 as shown in Fig. 15A.

[0098] As described above, the terminal device 1 uses the return frame 28 to return from the second area image 25 to the first area image 24. This allows the transmission of the second area image 25 to be continued even if the detection target 26 moves out of the second area frame 27 due to the behavior of the user 101 while the second area image 25 is displayed. For example, in a situation where it is desired to continuously deliver close-up images of an item as cropped images, the degree of freedom of the user 101's behavior can be increased. Furthermore, providing the return frame 28 also serves to prevent frequent switching of the transmitted image even if the detection target 26 frequently moves in and out of the second area frame 27 while the second area image 25 is being transmitted.

[0099] Next, an example will be described in which a detection frame 29 for switching to the second area image 25 is provided separately from the second area frame 27. Fig. 16A shows an example in which the detection frame 29 and the return frame 28 are set. Fig. 16A shows a state in which the detection target 26 is not within the detection frame 29, and the terminal device 1 uses the first area image 24 as the transmitted image.

[0100] Suppose that the behavior of the user 101 at a certain point in time causes the detection target 26 to enter the second area frame 27 on the captured image as shown in Fig. 16B. At this time, the terminal device 1 continues to send the first area image 24. Furthermore, when the terminal device 1 detects that the behavior of the user 101 at a certain point in time causes the detection target 26 to enter the detection frame 29 on the captured image as shown in Fig. 16C, the terminal device 1 sends the second area image 25 (e.g., a cropped image) as shown in Fig. 16D.

[0101] Thereafter, even if the terminal device 1 detects that the detection target 26 has moved out of the second area frame 27 on the captured image as shown in Fig. 16E due to the behavior of the user 101, the terminal device 1 maintains the sent image as the second area image 25. On the other hand, when the terminal device 1 detects that the detection target 26 has entered the return frame 28 on the captured image as shown in Fig. 16F due to the behavior of the user 101, the terminal device 1 performs control to return the sent image to the first area image 24 as shown in Fig. 16A.

[0102] The above is an example in which the detection frame 29 for switching to the second area image 25 and the return frame 28 for switching to the first area image 24 are set separately from the second area frame 27, thereby using the detection frame 29 and the return frame 28 as pseudo-switches. Note that although switching to the second area image 25 is not performed even when the detection target 26 enters the second area frame 27, it is also possible to perform switching when the detection target 26 enters the second area frame 27. In other words, this is an example in which, when the detection target 26 enters either the second area frame 27 or the detection frame 29 as shown in FIG. 16B, the second area image 25 is set as the sent image as shown in FIG. 16D. This allows the user 101 to switch to the second area image 25 without being limited only by the position of the second area frame 27.

[0103] Next, an example of the arrangement of the return frame 28 and the detection frame 29 will be described. Fig. 17A shows an example in which the first area image 24 is not the entire captured image of the camera 2, but rather a slightly narrower image indicated by the dashed line. For example, the user 101 sets the first area image 24 as an area that captures the entire scene but is not the entire captured image. Also, in this example, the second area frame 27 is a partial region within the first area image 24.

[0104] The return frame 28 and the detection frame 29 are set in an area outside the range of the first area image 24 within the image captured by the camera 2. In other words, the return frame 28 and the detection frame 29 are set at positions that are not included in either the first area image 24 or the second area frame 27.

[0105] The control using the detection frame 29 and return frame 28 is similar to the examples described in Figures 16A to 16F. That is, the terminal device 1 maintains the first area image 24 as the transmitted image even when the detection target 26 enters the second area frame 27 as shown in Figure 17B. When the detection target 26 enters the detection frame 29 as shown in Figure 17C, the terminal device 1 switches the transmitted image to the second area image 25 as shown in Figure 17D. Even when the detection target 26 moves out of the second area frame 27 as shown in Figure 17E, the terminal device 1 maintains the transmitted image to the second area image 25. When the detection target 26 enters the return frame 28 as shown in Figure 17F, the terminal device 1 switches the transmitted image to the first area image 24 as shown in Figure 17A.

[0106] In this example, the return frame 28 and the detection frame 29 are set at positions that are not included in the transmitted image, so that the viewer can be prevented from seeing the user 101's behavior of causing the detection target 26 to enter the return frame 28 or the detection frame 29. The terminal device 1 may set the range of the first area image 24 and the placement of the return frame 28 and the detection frame 29 in accordance with the setting operation of the user 101, for example, in step S130 of Fig. 7. In this case, the terminal device 1 may also switch the transmitted image to the second area image 25 as shown in Fig. 17D when the detection target 26 enters the second area frame 27 as shown in Fig. 17B.

[0107] Next, an example in which zoom processing is used instead of crop processing will be described. For example, it is assumed that one PTZ camera 2B is used.

[0108] 18A shows a state in which the detection target 26 is not within the second area frame 27, and the terminal device 1 sets the first area image 24 as the transmitted image. Assume that the terminal device 1 detects that the detection target 26 has entered the second area frame 27 on the captured image due to the behavior of the user 101 at a certain point in time, as shown in FIG. 18B. At this time, the terminal device 1 instructs the PTZ camera 2B to perform pan, tilt, and zoom operations on the second area frame 27. As a result, as shown in FIG. 18C, the second area image 25, which is a zoomed image of the range of the second area frame 27, is set as the transmitted image. Note that in FIG. 18C, the entire captured image of the PTZ camera 2B is shaded to indicate that the zoom angle of view has changed to an image corresponding to the area within the second area frame 27.

[0109] Suppose that at a certain point in time, due to the behavior of the user 101, the terminal device 1 detects that the detection target 26 has moved outside the second area frame 27 in the captured image, as shown in FIG. 18D. For example, this is the case when the detection target 26 can no longer be recognized in the captured image. At this time, the terminal device 1 instructs the PTZ camera 2B to perform pan / tilt / zoom operations to return to the original first area imaging state. As a result, the transmitted image returns to the first area image 24, as shown in FIG. 18A.

[0110] As described above, the terminal device 1 controls switching of the transmitted image between the first area image 24 (e.g., the entire image) and the second area image 25 (e.g., a zoomed image of a partial range of the captured image) depending on whether the detection target 26 is in or out of the second area frame 27. Using a zoomed image as the second area image 25 makes it possible to avoid degradation in image quality that occurs with cropped images.

[0111] Another advantage of using the PTZ camera 2B is that even if multiple second area frames 27 are set, the second area images 25 can be transmitted by performing pan, tilt, and zoom in accordance with each second area frame 27.

[0112] When zooming with one PTZ camera 2B, the zoom-in / zoom-out process is included in the transmitted image when switching between the first area image 24 and the second area image 25. This is not a technical problem, but if the user 101 wants to switch between the first area image 24 and the second area image 25 instantaneously, a method using two cameras 2 can be considered, as follows. For example, two cameras 2A are used, or a camera 2A and a PTZ camera 2B are used. Two PTZ cameras 2B are also acceptable.

[0113] 4, for example, camera 2A and PTZ camera 2B are installed close to each other, and camera 2A captures the entire scene. The image captured by camera 2A is designated as first area image 24. The pan / tilt position and zoom angle of view of PTZ camera 2B are set so that the image is captured within the range of a second area frame 27 set within the range of first area image 24. The image captured by PTZ camera 2B is designated as second area image 25.

[0114] The reason for installing the camera 2A and the PTZ camera 2B close to each other is to bring the viewpoint positions of the first area image 24 and the second area image 25 close to each other so that when switching between images, there is no sense of incongruity between the entire image and a partially enlarged image of the same scene. If the images are originally taken from different viewpoints as shown in Figure 3, there is no need to install the two cameras close to each other.

[0115] 19A shows a state in which the detection target 26 is not within the second area frame 27, and the terminal device 1 sets the first area image 24 captured by the camera 2A as the transmitted image. When the behavior of the user 101 at a certain point in time detects that the detection target 26 has entered the second area frame 27 on the captured image as shown in FIG. 19B, the terminal device 1 controls the second area image 25 captured by the PTZ camera 2B as the transmitted image. Note that while FIG. 19C shows the range of the captured image by the camera 2A, the second area image 25 within the shaded second area frame 27 is the entire captured image by the PTZ camera 2B.

[0116] Thereafter, if the behavior of the user 101 at a certain point in time detects that the detection target 26 has moved outside the second area frame 27 in the captured image as shown in Figure 19D, the terminal device 1 controls to switch the transmitted image to the first area image 24 captured by the camera 2A.

[0117] As described above, the terminal device 1 controls switching of the transmitted image between the first area image 24 (for example, the entire image) captured by the camera 2A and the second area image 25 (for example, a zoomed image) captured by the PTZ camera 2B, depending on whether the detection target 26 has entered or left the second area frame 27. In this case, switching can be performed without including the zoom process.

[0118] It should be noted that the PTZ camera 2B does not necessarily have to be used when switching between the first area image 24 and the second area image 25 in the transmitted image by switching between the images captured by the two cameras 2. This is because two cameras 2A may be used and the angle of view and shooting direction may be set so that each camera captures the first area image 24 and the second area image 25. However, the use of the PTZ camera 2B has the advantage that pan-tilt-zoom operations can be used even when multiple second area frames 27 are set.

[0119] Next, a display example of the skeleton points 32 on the terminal device 1 will be described. The terminal device 1 performs skeleton detection (skeleton estimation) from the image of the user 101, who is the subject, and determines the position of each body part. Therefore, it is possible to perform display based on the skeleton detection. Therefore, as shown in FIG. 13B , the skeleton points 32 are displayed on the image of the user 101 on the entire image display unit 21. In FIG. 13B , the skeleton points 32 are displayed when the skeleton display button 22 is turned on.

[0120] However, it is not necessary to always display the skeleton points 32. In both Fig. 20A and Fig. 20B, the skeleton display button 22 is turned on, but Fig. 20A shows an example in which the skeleton points 32 are not displayed, while Fig. 20B shows an example in which the skeleton points 32 are displayed. This is an example in which, for example, when the left wrist is set as the detection target 26 for the second area frame 27, the skeleton points 32 are displayed when the detection target 26 is within or near the second area frame 27.

[0121] Displaying the skeleton points 32 is useful for the user 101 to recognize the positional relationship between the body part selected as the detection target 26 and a frame such as the second area frame 27. However, this is less useful when the detection target 26 is far from the second area frame 27. The skeleton points 32 are particularly useful when the user 101 wants to confirm whether the body part selected as the detection target 26 is within the second area frame 27 or not. Therefore, as shown in Figures 20A and 20B, the skeleton points 32 are displayed when the detection target 26 is within or near the second area frame 27, and are not displayed otherwise. This prevents the screen from becoming cluttered by unnecessary display of the skeleton points 32.

[0122] 21A and 21B also show an example in which, when the skeleton display button 22 is turned on, skeleton points 32 are displayed when the left wrist, which is the detection target 26, is located within or near the second area frame 27. Moreover, this example displays only skeleton points 32 at or near the part that is the detection target 26. In Fig. 21B, only skeleton points 32 near the left wrist are displayed. This makes it possible to avoid cluttering the screen by displaying all skeleton points 32.

[0123] The skeleton points 32 may be displayed when the detection target 26 approaches a frame such as the second area frame 27 or when the detection target 26 is about to leave the second area frame 27, and the display may be turned off when the detection target 26 remains within the second area frame 27. Alternatively, the display may be turned off when the detection target 26 is stationary, and the display of the skeleton points 32 may be started when the part that becomes the detection target 26 starts to move.

[0124] Furthermore, by associating the skeleton points 32 indicating the parts of the detection target 26 with the display mode of the frame, it is possible to realize a display that is easy for the user 101 to understand.

[0125] 22A shows an example in which two second area frames 27A and 27B are set. Assume that the right wrist of the user 101 is set as the detection target 26A in the second area frame 27A, and the left wrist of the user 101 is set as the detection target 26A in the second area frame 27B. This is a setting in which, when the user 101 places his / her right wrist in the second area frame 27A, the image in the second area frame 27A becomes the sent image, and, when the user 101 places his / her left wrist in the second area frame 27B, the image in the second area frame 27B becomes the sent image.

[0126] In this case, the skeleton point 32A of the right wrist and the second area frame 27A are colored the same (e.g., red), and the skeleton point 32B of the left wrist and the second area frame 27B are colored the same (e.g., blue). This makes it easier for the user 101 to understand the relationship between the detection targets 26A and 26B and the second area frames 27A and 27B.

[0127] The correspondence may be indicated in a manner other than by color. For example, the correspondence may be indicated by the type of line (solid line, dashed line, dotted line, line width, etc.) that represents the second area frame 27 and the skeleton point 32. Furthermore, a display may be performed in which a line is drawn between the skeleton point 32 and the corresponding second area frame 27.

[0128] 22B shows an example in which two detection targets 26A (right wrist) and 26B (left wrist) are set with an AND condition for the second area frame 27. In other words, when both the right wrist and the left wrist enter the second area frame 27, the image in the second area frame 27 becomes the transmitted image.

[0129] In this case, the skeleton points 32A of the detection target 26A are colored red, the skeleton points 32B of the detection target 26B are colored blue, and the second area frame 27 is displayed as a double frame consisting of a red second area frame 27a and a blue second area frame 27b. This indicates that image switching is performed under the AND condition.

[0130] Figure 22C is also an example in which two detection targets 26A (right wrist) and 26B (left wrist) are set for the second area frame 27, and an OR condition is set such that if either the right wrist or the left wrist enters the second area frame 27, the image in the second area frame 27 becomes the transmitted image.

[0131] In this case, the skeleton points 32A of the detection target 26A are red, the skeleton points 32B of the detection target 26B are blue, and the second area frame 27 is colored in alternating red and blue, thereby indicating that the image switching is performed under the OR condition.

[0132] In the cases of FIGS. 22B and 22C, the correspondence relationship, AND condition, and OR condition may also be indicated by a display mode other than color.

[0133] However, when multiple second area frames 27 are set, the conditions for switching images may be satisfied simultaneously in multiple second area frames 27. For example, in the case of Fig. 22A, this may be the case when the right wrist is in second area frame 27A and the left wrist is in second area frame 27B. The following example is considered as an example of how to determine the image to be sent in such a case.

[0134] First, there is an example in which the priorities of the second area frames 27A and 27B are set in advance. For example, the terminal device 1 sets the priorities in response to the operation of the user 101 in the detailed setting of step S130 in Fig. 7. Then, for example, if the second area frame 27A is set to have priority, and both the second area frames 27A and 27B satisfy the conditions for switching images, the image of the second area frame 27A, which has priority, is set as the transmitted image.

[0135] Instead of setting a priority in advance, there is also a method in which the image that first satisfies the conditions takes priority. For example, if the second area frame 27A satisfies the conditions momentarily before the second area frame 27B, the image in the second area frame 27A is set as the image to be sent. Also, when the image in the second area frame 27A is set as the image to be sent, even if the second area frame 27B also satisfies the conditions, the image in the second area frame 27A continues to be set as the image to be sent. This makes it possible to prevent frequent switching of the image to be sent and confusion for the user 101.

[0136] Furthermore, if the image switching conditions are met for multiple second area frames 27, the user 101 may be notified. Fig. 23A shows a state in which second area frames 27A and 27C are set with the right wrist as the detection target 26A, and second area frame 27B is set with the left wrist as the detection target 26B. Assume now that the right wrist is within second area frame 27A. In this case, the conditions for second area frame 27A are met, and the image in second area frame 27A is set as the transmitted image. The fact that the conditions for second area frame 27A are currently met is indicated by a thick frame.

[0137] After that, as shown in Fig. 23B, it is assumed that the condition of the second area frame 27B is also satisfied. At this time, both the second area frame 27A and the second area frame 27B are in a condition of satisfying the condition. In this case, the second area frame 27A, which satisfied the condition first, is given priority as the sent image, and a message 33 is displayed as shown in Fig. 23C, and the second area frame 27A and the second area frame 27B are displayed in a flashing manner. In this way, by making the user 101 aware of the behavior of satisfying multiple frames simultaneously, confusion of the user 101 when multiple frames are satisfied simultaneously can be prevented.

[0138] 5. Configuration of Each Embodiment Below, examples of functional configurations of the terminal device 1 adapted to various system configurations will be described as first to fifth embodiments.

[0139] First Embodiment First, an example of capturing images using one camera 2A will be described, as shown in Fig. 24. When capturing images is started, the camera 2A transmits captured images in intermittent frames (hereinafter referred to as intermittent images P2) to the terminal device 1, and also transmits full-frame captured images (hereinafter referred to as stream images P1) that are to be used as actual video content to the terminal device 1.

[0140] The intermittent image P2 is, for example, JPEG (Joint Photographic Experts Group) image data, which is used by the terminal device 1 to detect the skeleton of the user 101, who is the subject. The stream image P1 is, for example, FHD (Full High Definition) image data. The stream image P1 sent to the terminal device 1 may be a first area image 24 representing the entire image captured by the camera 2A, or a second area image 25 cut out by cropping from the image captured by the camera 2A.

[0141] The terminal device 1 transmits a control signal CS1 to the camera 2A. The control signal CS1 includes a crop instruction signal. The terminal device 1 uploads the stream image P1 from the camera 2A to the distribution server 3 as a transmission image P1S.

[0142] 25 shows the functional configuration provided in the camera 2 and the terminal device 1 in this case. The camera 2 includes an image input unit 40, an audio input unit 41, a detected image sending unit 42, an image processing unit 43, and a streaming sending unit 44.

[0143] The image input unit 40 includes a lens system, an image sensor, a photoelectric conversion signal processor, a digital signal processor, etc., and photoelectrically converts subject light and then converts it into digital signals to obtain image data. Then, preprocessing, brightness processing, color processing, etc. are performed to generate image data as a captured image. The audio input unit 41 includes a microphone and an audio circuit that amplifies collected audio signals, etc., and outputs audio data to be added to the image stream P1.

[0144] The image processing unit 43 performs various signal processing to generate the stream image P1, such as image effect processing, resolution conversion processing, and encoding processing, on the image data from the image input unit 40. For example, the image processing unit 43 also performs cropping processing to cut out the range of the second area frame 27.

[0145] The detected image sending unit 42 acquires image data from the image input unit 40 in intermittent frames, performs JPEG encoding, and sends the data to the terminal device 1. The streaming sending unit 44 forms a stream image P1 using the image data from the image processing unit 43 and the audio data from the audio input unit 41, and sends the stream image P1 to the terminal device 1.

[0146] The terminal device 1 includes a determination / instruction unit 50, a drawing unit 51, a rotation setting unit 52, a rotation processing unit 53, a skeleton estimation unit 54, a frame setting unit 55, and a rotation processing unit 56. These are functions realized by a CPU 71 in accordance with a program in an information processing device 70 shown in FIG. 5 , which is the terminal device 1.

[0147] The rotation setting unit 52 is a function for setting the rotation process of an image in response to an operation input by the user 101 via the input unit 76. For example, when the camera 2 outputs a captured image in landscape orientation, if the video content is to be a portrait image, the rotation setting unit 52 sets the vertical / horizontal rotation of the image. In the following, an example will be described in which the video content is produced as a portrait image.

[0148] The rotation processing unit 53 performs processing to rotate the interval image P2 transmitted from the camera 2A into a vertically long image. The skeleton estimation unit 54 performs skeleton estimation on the interval image P2 from the rotation processing unit 53, for example, by AI processing, and detects the skeleton position of the user 101 who is the subject.

[0149] The frame setting unit 55 performs the above-described setting process and stores the set information. For example, it stores the settings of the second area frame 27 and the detection target 26. Furthermore, it performs the above-described effect setting, priority setting, satisfying condition setting, personal setting, setting of the return frame 28 and the detection frame 29, and setting to prevent the first area from becoming the entire image as shown in FIG. 17 in accordance with user input, and stores the settings.

[0150] The determination and instruction unit 50 has a function of determining whether conditions for the second area frame 27, etc. are satisfied and performing control in accordance with the determination. The determination and instruction unit 50 acquires skeleton detection information from the skeleton estimation unit 54, and also acquires information on the detection target 26 and the second area frame 27, as well as various other setting information, from the frame setting unit 55. The determination and instruction unit 50 then determines the position of the detection target 26 on the intermittent image P2 from the rotation processing unit 53 based on the skeleton detection information, and determines the positional relationship between the detection target 26 and the second area frame 27, etc. In other words, it performs a determination as to whether the conditions for switching the transmitted image described with reference to FIGS. 14A to 19D are satisfied. The determination and instruction unit 50 then transmits a control signal CS1 to the image processing unit 43 of the camera 2A in accordance with the result of the determination as to whether the conditions are satisfied. This control signal CS1 serves as an instruction to crop the range of the second area frame 27.

[0151] The rotation processing unit 56 rotates the stream image P1 transmitted from the camera 2A into a vertically long image. The output of the rotation processing unit 56 is used as the transmission image P1S to the distribution server 3.

[0152] 12A to 23C, the drawing unit 51 performs drawing processing on the monitor screen 15. To this end, the drawing unit 51 inputs images (images before or after rotation processing) from the rotation processing units 53 and 56, skeleton detection results from the skeleton estimation unit 54, frame setting information from the frame setting unit 55, and condition satisfaction determination results from the determination / instruction unit 50, and controls the display on the monitor screen 15.

[0153] The distribution server 3 includes a streaming receiving unit 100, which receives the transmitted image P1S and performs distribution processing to the viewer.

[0154] With this configuration, depending on whether the conditions in the relationship between the second area frame 27 (or further the return frame 28 or the detection frame 29) and the detection target 26 are met, the image P1S sent from the stream image P1 will be either the first area image 24 or the second area image 25 through crop processing.

[0155] 26 , the second embodiment is also an example in which shooting is performed using one camera 2A. However, in this example, cropping is performed within the terminal device 1. When shooting starts, the camera 2A transmits a stream image P1, which is to be used as actual video content, to the terminal device 1.

[0156] The terminal device 1 uses the stream image P1 to perform skeleton detection, condition satisfaction determination, cropping processing, etc., and as a result, generates a transmission image P1S in which the first area image 24 and the second area image 25 are sequentially switched, and uploads it to the distribution server 3.

[0157] The functional configuration provided in the camera 2 and the terminal device 1 in this case is shown in Fig. 27. The camera 2 is provided with an image input unit 40, an audio input unit 41, an image processing unit 43, and a streaming sending unit 44, as in Fig. 25. However, the detected image sending unit 42 shown in Fig. 25 does not need to be provided. Also, there is no need to perform cropping in the image processing unit 43.

[0158] 25, the terminal device 1 includes an image processing unit 57 and an image extraction unit 58. These are also functions realized by a CPU 71 in accordance with a program in the information processing device 70 of FIG. 5, which is the terminal device 1.

[0159] The image extraction unit 58 extracts intermittent frames from the stream image P1 and outputs an intermittent image P2. This intermittent image P2 is rotated by the rotation processing unit 53 and supplied to the skeleton estimation unit 54 for use in skeleton detection. The image extraction unit 58 may also extract frames from the stream image P1 and further convert them into low-resolution images. This reduces the processing load on the skeleton estimation unit 54.

[0160] The image processing unit 57 has a function of performing crop processing on the stream image P1. The image processing unit 57 performs crop processing within the range from the second area frame 27 in accordance with a control signal CS1 from the determination / instruction unit 50. The stream image P1 that has not been cropped by the image processing unit 57 or the stream image P1 that has been cropped is rotated by the rotation processing unit 56 to become a sent image P1S.

[0161] With the above configuration, depending on whether the conditions in the relationship between the second area frame 27, etc. and the detection target 26 are met, the transmitted image P1S is either the first area image 24 or the second area image 25 obtained by cropping.

[0162] Third Embodiment In the third embodiment, as shown in FIG. 28, an example is shown in which photography is performed using one PTZ camera 2B.

[0163] When shooting starts, the PTZ camera 2B transmits the intermittent image P2 to the terminal device 1 and also transmits the stream image P1 to the terminal device 1.

[0164] The terminal device 1 transmits a control signal CS2 to the PTZ camera 2B. The control signal CS2 includes commands for panning, tilting, and zooming. The terminal device 1 uploads the stream image P1 from the PTZ camera 2B to the distribution server 3 as a transmitted image P1S.

[0165] Fig. 29 shows the functional configuration provided in the PTZ camera 2B and the terminal device 1 in this case. The PTZ camera 2B is provided with an image input unit 40, an audio input unit 41, a detected image sending unit 42, an image processing unit 43, and a streaming sending unit 44, similar to Fig. 25. However, there is no need to perform cropping in the image processing unit 43.

[0166] Additionally, the PTZ camera 2B is provided with a PTZ control unit 45. The PTZ control unit 45 has a function of controlling the driving of the pan mechanism, tilt mechanism, and zoom mechanism in the PTZ camera 2B.

[0167] The configuration of the terminal device 1 in Fig. 29 is the same as that in Fig. 25. However, the difference is that the determination / instruction unit 50 transmits a control signal CS2 to the PTZ camera 2B depending on the result of the condition satisfaction determination. As a result, the PTZ control unit 45 performs the necessary pan / tilt / zoom operations, and the PTZ camera 2B outputs a stream image P1 as the second area image 25.

[0168] With the above configuration, depending on whether the conditions in the relationship between the second area frame 27 etc. and the detection target 26 are met, the transmitted image P1S is set to the first area image 24 or the second area image 25 by zooming.

[0169] [Fourth embodiment] In the fourth embodiment, as shown in Fig. 30, an example is taken in which two cameras 2A are used to take pictures. The cameras 2A are designated as cameras 2A1 and 2A2. For example, a scene as shown in Fig. 3 is assumed.

[0170] For example, camera 2A1 captures an image of a first area (e.g., an entire image of a cooking scene) and generates stream image P11, which becomes first area image 24. Camera 2A2 captures an image of a second area (e.g., a partial image of a close-up of the food) and generates stream image P12, which becomes second area image 25. When shooting begins, cameras 2A1 and 2A2 transmit stream images P11 and P12, respectively, to terminal device 1.

[0171] The terminal device 1 uses the stream images P11 and P12 to perform skeleton detection, condition satisfaction determination, switcher processing, etc., and as a result, generates a transmission image P1S in which the first area image 24 and the second area image 25 are sequentially switched, and uploads it to the distribution server 3.

[0172] 31 shows the functional configuration provided in cameras 2A1 and 2A2 and terminal device 1 in this case. Cameras 2A1 and 2A2 each include an image input unit 40, an audio input unit 41, an image processing unit 43, and a streaming sending unit 44. The detected image sending unit 42 is not required, and the image processing unit 43 does not need to perform cropping processing.

[0173] 25, the terminal device 1 includes a switcher unit 59 and an image extraction unit 58. These are also functions realized by a CPU 71 in accordance with a program in the information processing device 70 of FIG. 5, which is the terminal device 1.

[0174] The switcher unit 59 has a video switcher function of selecting and outputting either the stream image P11 or P12. Either the stream image P11 or P12 selected by the switcher unit 59 is rotated by the rotation processing unit 56 to become the transmitted image P1S.

[0175] The image extraction unit 58 extracts intermittent frames from each of the stream images P11 and P12 and outputs intermittent images P21 and P22. These two systems of intermittent images P21 and P22 are each rotated by the rotation processing unit 53 and supplied to the skeleton estimation unit 54 for use in skeleton detection. Note that the image extraction unit 58 may extract frames from the stream images P11 and P12 and further convert them into low-resolution images in order to reduce the processing load on the skeleton estimation unit 54. Furthermore, although not separately shown, the intermittent images P21 and P22 may be supplied from the cameras 2A1 and 2A2, respectively, and input to the rotation processing unit 53. In this case, the image extraction unit 58 is unnecessary.

[0176] The skeleton estimation unit 54 can detect the skeleton for each of the interval images P21 and P22. The determination and instruction unit 50 determines whether the conditions are met based on the skeleton detection result and the second area frame 27. Based on the result, the determination and instruction unit 50 transmits a control signal CS3 to the switcher unit 59. The switcher unit 59 performs switching based on the control signal CS3.

[0177] With the above configuration, depending on whether the conditions in the relationship between the second area frame 27 etc. and the detection target 26 are met, the transmitted image P1S is either the first area image 24 taken by camera 2A1 or the second area image 25 taken by camera 2A2.

[0178] In this example, skeleton detection is performed on each of the interval images P21 and P22. This makes it possible to set a second area frame 27 for each of the images captured by cameras 2A1 and 2A2 and perform condition satisfaction determination separately. For example, in a captured scene such as that shown in FIG. 3, it is possible to perform condition satisfaction determination based on the positional relationship between the second area frame 27 and the detection target 26 in a front view taken by camera 2A1, and also to perform condition satisfaction determination based on the positional relationship between the second area frame 27 and the detection target 26 in a planar view taken by camera 2A2. In other words, it is possible to perform condition satisfaction determination based on a three-dimensional positional relationship and switch images accordingly.

[0179] 31 , skeleton detection may be performed using only the interval image P21, and the condition fulfillment determination may be made based on the positional relationship between the second area frame 27 and the detection target 26 on the image captured by camera 2A1. Alternatively, skeleton detection may be performed using only the interval image P22, and the condition fulfillment determination may be made based on the positional relationship between the second area frame 27 and the detection target 26 on the image captured by camera 2A2.

[0180] Fifth Embodiment In the fifth embodiment, as shown in FIG. 32, an example is shown in which photography is performed using a camera 2A and a PTZ camera 2B.

[0181] When shooting starts, the camera 2A and the PTZ camera 2B transmit stream images P1A and P1B to the terminal device 1, respectively.

[0182] The terminal device 1 performs skeleton detection, condition fulfillment determination, switcher processing, etc. using, for example, the stream image P1A, and as a result, generates a transmission image P1S in which the first area image 24 and the second area image 25 are sequentially switched, and uploads it to the distribution server 3.

[0183] In this case, the functional configuration provided in camera 2A, PTZ camera 2B, and terminal device 1 is shown in Fig. 33. Camera 2A and PTZ camera 2B each include an image input unit 40, an audio input unit 41, an image processing unit 43, and a streaming transmission unit 44. PTZ camera 2B also includes a PTZ control unit 45. Terminal device 1 has the same configuration as that shown in Fig. 31.

[0184] The switcher unit 59 selects and outputs either the stream image P1A or P1B. Either the stream image P1A or P1B selected by the switcher unit 59 is rotated by the rotation processing unit 56 to become a transmission image P1S.

[0185] The image extraction unit 58 extracts intermittent frames from the stream image P1A and outputs an intermittent image P2A. This intermittent image P2A is rotated by the rotation processing unit 53 and supplied to the skeleton estimation unit 54 for use in skeleton detection.

[0186] The determination and instruction unit 50 determines whether the conditions are met based on the skeleton detection result and the second area frame 27. Based on the result, the determination and instruction unit 50 sends a control signal CS3 to the switcher unit 59. The switcher unit 59 performs switching based on the control signal CS3. Furthermore, based on the result of the condition fulfillment determination, the determination and instruction unit 50 sends a control signal CS2 to the PTZ control unit 45 of the PTZ camera 2B as necessary. This causes the PTZ camera 2B to perform pan, tilt, and zoom operations.

[0187] With the above configuration, depending on whether the conditions in the relationship between the second area frame 27 etc. and the detection target 26 are met, the transmitted image P1S is either the first area image 24 taken by the camera 2A1 or the second area image 25 taken by the PTZ camera 2B.

[0188] By using the PTZ camera 2B, even if multiple second area frames 27 are set, second area images 25 corresponding to each second area frame 27 can be obtained as stream images P1B by pan / tilt / zoom operations.

[0189] 6. Example of Transmission Image Change Control Processing> An example of a transmission image change control processing performed by the terminal device 1 in the configuration of each of the above-described embodiments will be described. Each of the following flowcharts is processing executed by the CPU 71 of the information processing device 70, which is the terminal device 1, by a program that has the functions shown in the first to fifth embodiments. Note that the same processing as in the flowcharts already described will be assigned the same step numbers, and redundant explanations will be avoided.

[0190] 34 shows an example of processing performed when producing video content, such as live streaming, by the terminal device 1. The figure shows the processing of transmitted image change control for switching between the first area image 24 and the second area image 25 during the period from the start of transmission of the captured transmitted image P1S to the end of transmission.

[0191] When shooting starts, the terminal device 1 performs control in step S201 to set the first area image 24 as the sent image P1S. Specifically, the terminal device 1 performs the following process in each configuration example.

[0192] In the configuration example of Fig. 25, the terminal device 1 outputs a control signal CS1 to the camera 2 instructing not to perform cropping, and starts processing to output the stream image P1 as a sent image P1S. In the configuration example of Fig. 27, the determination / instruction unit 50 of the terminal device 1 outputs a control signal CS1 instructing not to perform cropping, and the image processing unit 57 starts processing to output the stream image P1 that has not been subjected to cropping, as a sent image P1S.

[0193] In the configuration example of Fig. 29, the terminal device 1 transmits a control signal CS2 to the PTZ camera 2B to instruct the PTZ camera 2B to pan, tilt, and zoom to capture the first area image 24, and starts processing to output the stream image P1 as the transmitted image P1S. In the configuration example of Fig. 31, the determination and instruction unit 50 of the terminal device 1 outputs a control signal CS3 to the switcher unit 59 to instruct the switcher unit 59 to select the stream image P11 of the camera 2A1, and starts processing to output the stream image P11 as the transmitted image P1S. In the configuration example of Fig. 33, the determination and instruction unit 50 of the terminal device 1 outputs a control signal CS3 to the switcher unit 59 to instruct the switcher unit 59 to select the stream image P1A of the camera 2A, and starts processing to output the stream image P1A as the transmitted image P1S.

[0194] Thereafter, the terminal device 1 repeats the process of step S202 in FIG. 34 and the monitoring process of steps S203, S204, S205 or S206, S207.

[0195] In step S202 of Fig. 34, the terminal device 1 performs monitor screen control, thereby starting the display on the monitor screen 15 as shown in Fig. 12A and Fig. 12B. Detailed processing examples of step S202 are shown in Fig. 35 and Fig. 36.

[0196] 35, first, in step S220, the terminal device 1 starts processing to display the captured image on the entire image display unit 21 and display the sent image P1S on the sent image display unit 20. In step S221, the terminal device 1 starts frame display on the entire image display unit 21. That is, the second area frame 27 is displayed. In some cases, a return frame 28 or a detection frame 29 may be displayed.

[0197] In step S222, the terminal device 1 checks the setting based on the operation of the skeleton display button 22, and checks whether the display setting of the skeleton points 32 is currently on or off. If it is off, the terminal device 1 turns off the display of the skeleton points 32 in step S224. In other words, if the display had been on until then, the display is ended, and if the display had been off until then, the display remains off. Then, the processing of step S202 is completed.

[0198] If the setting based on the operation of the skeleton display button 22 is ON, the terminal device 1 proceeds to step S223, and starts displaying the skeleton points 32 superimposed on the user 101 in the whole image display section 21. Then, the processing of step S202 ends.

[0199] The terminal device 1 may display all of the detected skeleton points 32, or may display only some of the skeleton points 32. The terminal device 1 also displays the skeleton points 32 in a display manner (e.g., color) such that the skeleton points 32 corresponding to the detection target 26 are displayed in the same color as the corresponding second area frame 27, thereby indicating the correspondence. In particular, when a plurality of second area frame 27 are provided, the correspondence between the second area frame 27 and the skeleton points 32 of the detection target 26 is indicated by matching their colors, etc.

[0200] 36 shows an example in which the processing in step S202 is different from the display control of the skeleton points 32. Steps S220, S221, and S222 are the same as those in Fig. 35. If the display setting of the skeleton points 32 is currently on, the terminal device 1 proceeds to step S225, where the processing branches depending on whether the currently transmitted image P1S is the first area image 24 or the second area image 25.

[0201] If the sent image P1S is the first area image 24, the terminal device 1 proceeds to step S226 and determines whether or not the skeleton points 32 of the detection target 26 are in the vicinity of the second area frame 27. This determination can be made by determining whether or not they are within a predetermined distance from the second area frame 27. If it is determined that they are in the vicinity, the terminal device 1 proceeds to step S223 and starts displaying the skeleton points 32. If it is not determined that they are in the vicinity, the terminal device 1 proceeds to step S224 and turns off the display of the skeleton points 32.

[0202] If the currently sent image P1S is the second area image 25, the terminal device 1 proceeds from step S225 to step S227, where it determines whether the skeleton points 32 of the detection target 26 are near the boundary within the second area frame 27. This determination can be made by determining whether they are within a predetermined distance from the frame line within the second area frame 27. If it is determined that they are near, the terminal device 1 proceeds to step S223, where it starts displaying the skeleton points 32. If it is not determined that they are near, the terminal device 1 proceeds to step S224, where it turns off the display of the skeleton points 32.

[0203] When the terminal device 1 displays the skeleton points in step S223, there are examples in which all of the detected skeleton points 32 are displayed as shown in Fig. 20B , and examples in which only some of the skeleton points 32 are displayed as shown in Fig. 21B . Also in this case, the terminal device 1 may display the skeleton points 32 (for example, in color) in such a way that the skeleton points 32 corresponding to the detection target 26 are displayed in the same color as the corresponding second area frame 27, thereby indicating the correspondence. This is particularly suitable when there are multiple second area frames 27.

[0204] After performing the processing in each of the above examples in step S202 of Fig. 34, the terminal device 1 proceeds to step S203, where it monitors the display operation of the skeleton points 32. This is a check of the operation of the skeleton display button 22. If the user 101 operates the skeleton display button 22, it proceeds to step S208, where it switches the display setting of the skeleton points 32 on / off in accordance with the operation. When it returns to step S202, the display of the skeleton points 32 is switched on / off in accordance with this setting.

[0205] In step S204, the terminal device 1 branches the process depending on whether or not the first area image 24 is being transmitted. If the transmitted image P1S is the first area image 24, the terminal device 1 proceeds to step S205 and determines whether or not the conditions for switching from the first area image 24 to the second area image 25 are satisfied.

[0206] 37, 38, and 39 show examples of the condition fulfillment determination in step S205. In the example of Fig. 37, the terminal device 1 determines in step S251 whether or not the detection target 26 is located within the second area frame 27. If a detection frame 29 is provided, it determines whether or not the detection target 26 is located within the frame of the detection frame 29.

[0207] If it is determined that the detection target 26 is within the second area frame 27 or the detection frame 29, the terminal device 1 checks the duration of stay within the frame in step S252 and determines whether the duration of stay within the frame is equal to or longer than a predetermined time. The predetermined time can be set to, for example, about 1 to 2 seconds.

[0208] If the duration of stay within the frame of the detection target 26 is equal to or longer than a predetermined time, the terminal device 1 determines that the target 26 is within the frame and satisfies the condition of duration of stay within the frame, i.e., the condition is satisfied, and proceeds to step S209. On the other hand, if a negative result is obtained in step S251 or step S252, the terminal device 1 determines that the condition is not satisfied and proceeds to step S207.

[0209] 38 is an example in which step S253 is added to FIG. 37 , and is an example that can be applied to the following cases, for example: - A case in which two second area frames 27a, 27b for the same area but different detection targets 26 are set and an AND condition is set, as shown in FIG. 22B . - A case in which two second area frames 27a, 27b for the same area but different detection targets 26 are set and an OR condition is set, as shown in FIG. 22C . - A case in which one second area image 25 and one detection frame 29 are provided and an OR condition is set that the detection target 26 enters either of the frames. - A case in which entry into the second area image 25 does not satisfy the condition, and multiple detection frames 29 are provided and an OR condition is set that the detection target 26 enters either of the detection frames 29, or an AND condition is set that the detection target 26 enters both of the detection frames 29.

[0210] The above are merely examples, but in these cases, if the terminal device 1 determines in steps S251 and S252 of FIG. 38 that the detection target 26 has been within a certain second area frame 27 (or detection frame 29) for a predetermined period of time or longer, the terminal device 1 checks whether the AND condition or the OR condition is satisfied in step S253. If the OR condition is satisfied for a frame in which the corresponding detection target 26 has been confirmed to have been within the frame for a predetermined period of time or longer, the condition is determined to be satisfied. In the case of the AND condition, if the corresponding detection target 26 has also been confirmed to have been within the frame for a predetermined period of time or longer in other frames, the condition is determined to be satisfied. In these cases, the terminal device 1 proceeds to step S209.

[0211] On the other hand, if a negative result is obtained in step S251, step S252, or step S253, the terminal device 1 determines that the condition is not satisfied and proceeds to step S207.

[0212] 39A shows an example of processing in which a plurality of second area frames 27 that can become second area images 25 are set as in FIG. 22A , and a message 33 is output when conditions are detected to be satisfied for a plurality of second area frames 27, as described in FIG. 23C . When a plurality of second area frames 27 that can become second area images 25 are set, the processing in FIG. 34 is partially modified as shown in FIG. 39B . That is, step S205A is inserted between step S204 and step S206. This step S205A corresponds to the processing in FIG. 39A . Therefore, even when it is determined in step S204 of FIG. 34 that the first area image 24 is not being transmitted, that is, even when the second area image 25 for a certain second area frame 27 is being transmitted, the second area condition satisfaction determination is performed for the other second area frame 27 in step S205A of FIG. 39B . Thereafter, the process proceeds to step S206, where it is determined whether the release condition is met for the second area frame 27 on the side currently used as the sent image P1S.

[0213] The processing of step S205A in FIG. 39B in such an example case will be described with reference to FIG. 39A. If the terminal device 1 determines in steps S251 and S252 that the detection target 26 has stayed within a second area frame 27 that is not currently the area of ​​the transmitted image P1S for a predetermined period of time or longer, it checks in step S255 whether the condition is satisfied for multiple second area frames 27. If the condition is satisfied only for the second area frame 27 that is currently determined to be satisfied, the condition is satisfied for that second area frame 27 and the process proceeds to step S209. On the other hand, if the condition is satisfied for multiple second area frames 27, the terminal device 1 proceeds to step S256, where it displays a message or flashes the frame, as shown in FIG. 23C. Then, the process proceeds to step S206. In this case, it is assumed that the condition has already been satisfied for one of the second area frames 27 and that the second area image 25 has been transmitted. Therefore, if the conditions are satisfied for multiple second area frames 27, the second area image 25 that satisfied the conditions first will continue to be sent.

[0214] 37, 38, and 39, if it is determined that the condition is satisfied, the terminal device 1 proceeds from step S205 to step S209 in Fig. 34, and performs control processing to switch the sent image P1S to the second area image 25. A specific example will be described later. If it is not determined that the condition is satisfied, the terminal device 1 proceeds from step S205 to step S207 in Fig. 34, and returns to step S202 if the sending of the sent image P1S has not been completed.

[0215] 34, if the sent image P1S is the second area image 25, the terminal device 1 proceeds to step S206 and determines whether the cancellation condition for the second area frame 27 is met. This is a determination as to whether the condition for canceling the state in which the sent image P1S has been changed to the second area image 25 is met. The same applies when proceeding to step S206 in the modified example of FIG. 39B.

[0216] Examples of the determination of whether the cancellation condition is satisfied in step S206 are shown in Figures 40 and 41. In the example of Figure 40, the terminal device 1 determines whether the detection target 26 is located outside the second area frame 27 in step S261.

[0217] If it is determined that the detection target 26 is outside the second area frame 27, the terminal device 1 determines in step S262 whether the time during which the detection target 26 has been outside the frame is equal to or longer than a predetermined time. The predetermined time may be, for example, about 1 to 2 seconds.

[0218] If the time that the detection target 26 remains outside the frame is equal to or longer than a predetermined time, the terminal device 1 determines that the detection target 26 has escaped from the second area frame 27, determines that the release condition is met, and proceeds to step S201. In step S201, as described above, the terminal device 1 performs control to change the first area image 24 to the sent image P1S. That is, in this case, the sent image P1S is returned from the second area image 25 to the first area image 24.

[0219] On the other hand, if a negative result is obtained in step S261 or step S262, the terminal device 1 determines that the cancellation condition is not satisfied and proceeds to step S207.

[0220] 41 shows an example in which a return frame 28 is provided. In step S263, the terminal device 1 determines whether the detection target 26 is located within the return frame 28.

[0221] If it is determined that the detection target 26 is within the return frame 28, the terminal device 1 determines in step S264 whether the time during which the detection target 26 has been within the frame is equal to or longer than a predetermined time, which may be, for example, one to two seconds.

[0222] If the detection target 26 stays within the frame for a predetermined time or longer, the terminal device 1 determines that the detection target 26 has entered the return frame 28, determines that the release condition is met, and proceeds to step S201. As a result, the sent image P1S is returned from the second area image 25 to the first area image 24. On the other hand, if a negative result is obtained in step S263 or step S264, the terminal device 1 determines that the release condition is not met and proceeds to step S207.

[0223] Up to this point, examples of the processes for determining whether a condition is satisfied and whether a condition is released have been given, but other processes are also possible. For example, there may be multiple people as the subject. In that case, if a person's "left wrist" is set as the detection target 26, it may be possible to select whether to set the left wrist of a specific person or anyone. If anyone is acceptable, the processes shown in FIGS. 37, 38, and 39A may be used.

[0224] On the other hand, when setting the detection target 26 for a specific individual, for example, the face photo and skeleton of the person are registered during the setting process. Then, when photographing, if the skeleton is detected, the person's face is recognized and a match with the pre-registered face is confirmed. If a match is confirmed, a condition fulfillment determination, a release condition fulfillment determination, etc. are performed based on the relationship between the detection target 26 and the second area frame 27. By doing this, even if there is an actor acting as an assistant, for example, a specific user 101 can switch the transmitted image P1S.

[0225] 42 to 46 will be used to explain the process of proceeding from step S205 to step S209 in Fig. 34 (or Fig. 39B) and switching the sent image P1S from the first area image 24 to the second area image 25. However, the following example may also be performed when proceeding from step S205A to step S209 as in Fig. 39B, that is, when multiple second area frames 27 are set and switching from one second area image 25 to another second area image 25.

[0226] Fig. 42 shows a processing example for the configuration example of Fig. 25. In step S291, the terminal device 1 outputs a control signal CS1 to the camera 2, which instructs the camera 2 to perform cropping and to specify the cropping range (the range of the second area frame 27 for which the conditions have been satisfied). As a result, the stream image P1 becomes a cropped image, and the resulting cropped image is used as the transmitted image P1S.

[0227] Fig. 43 shows an example of processing in the configuration example of Fig. 27. In step S292, the determination and instruction unit 50 of the terminal device 1 sets cropping processing using the control signal CS1, and in step S293, the image processing unit 57 starts cropping processing within the range of the second area frame 27 for which it has determined that the conditions are met. As a result, the stream image P1 becomes a cropped image, and the cropped image is used as the transmitted image P1S.

[0228] Fig. 44 shows an example of processing in the case of the configuration example of Fig. 29. In step S295, the terminal device 1 transmits to the PTZ camera 2B a control signal CS2 instructing the PTZ camera 2B to pan, tilt, and zoom in order to capture a second area image 25, which is an image within the range of the second area frame 27 for which it has been determined that the conditions are met. This causes the PTZ camera 2B to perform pan, tilt, and zoom operations, and the stream image P1 becomes the second area image 25, which is then used as the transmitted image P1S.

[0229] Fig. 45 shows a processing example for the configuration example of Fig. 31. In step S294, the switcher unit 59 of the terminal device 1 selects the stream image P12 of the camera 2A2 in response to the control signal CS3. In this case, the stream image P12 is an image within the range of the second area frame 27 for which the condition has been determined to be satisfied. Therefore, the second area frame 27 is switched to a state in which it is used as the transmitted image P1S.

[0230] 46 shows a processing example for the configuration example of FIG. 33. In step S295, the terminal device 1 uses a control signal CS2 to instruct the PTZ camera 2B to pan, tilt, and zoom to capture the second area image 25, which is within the range of the second area frame 27 for which the conditions have been determined to be met. This causes the PTZ camera 2B to perform pan, tilt, and zoom operations. In step S296, the terminal device 1 waits for the pan, tilt, and zoom operations to be completed, and upon completion, in step S297, the switcher unit 59 uses a control signal CS3 to select the stream image P1B of the PTZ camera 2B. This switches the state so that the second area image 25 is the transmitted image P1S. By switching in the switcher unit 59 after the pan, tilt, and zoom operations are completed, the image in the zoom process is not included in the transmitted image P1S.

[0231] By performing the processing of step S209 in Fig. 34 in any of the above ways, the second area image 25 is switched to a state where it is set as the sent image P1S. The terminal device 1 repeats the processing of Fig. 34 until it is determined in step S207 that the sending of the sent image P1S has ended. As a result, the first area image 24 and the second area image 25 are automatically switched in accordance with the determination of whether the condition is met and whether the release condition is met. Furthermore, there are also cases where multiple second area frames 27 are set, and the first area image 24 and the multiple second area images 25 are automatically switched in accordance with the determination of whether the condition is met and whether the release condition is met.

[0232] Next, frame adjustment will be described. In the process described so far, the second area image 25 is an image within the range of the second area frame 27 in terms of image content. Here, the second area frame 27 is a frame that is set in advance. Therefore, an item 6 or the like that the user 101 wants to show up close during shooting may not necessarily fit neatly within the second area image 25.

[0233] For example, Fig. 47A shows an example of a display of the whole image display section 21 on the monitor screen 15 of Fig. 12A. Assume that the second area image 25 shown by the dotted portion is the sent image P1S. Fig. 47A shows a state in which the user 101 wanted to show an item 6 held in the palm of his / her hand, but the item 6 was not properly included in the second area image 25. For example, the top of the item 6 is cut off in the second area image 25.

[0234] In such a case, for example, it is preferable to move the second area frame 27 slightly upward as shown in Fig. 47B so that the item 6 is more appropriately captured in the second area image 25. Alternatively, it is preferable to enlarge the second area frame 27 as shown in Fig. 47C so that the item 6 is more appropriately captured in the second area image 25.

[0235] As illustrated above, when the range of the second area frame 27 is distributed as the second area image 25, it may be desirable to move the second area frame 27 in the up, down, left, and right directions, or to enlarge or reduce it depending on the subject. Therefore, the terminal device 1 performs the adjustment process shown in Fig. 48. For example, when the second area image 25 is set as the sent image P1S, the terminal device 1 performs the process shown in Fig. 48.

[0236] In step S301, the terminal device 1 performs object recognition of the subject. This is a process of determining the subject item through image analysis of the second area image 25. In step S302, the terminal device 1 determines the reflection state of the recognized item in the second area image 25. For example, the terminal device 1 determines the appropriateness of the reflection, such as whether the entire item is captured, or even if not the entire item, whether a key part is captured. In this case, it is advisable to determine whether the reflection is appropriate by taking into account the relationship with the body parts of the user 101, such as whether the user 101 is holding the item 6 in the palm of their hand, holding it, pinching it from above, or pointing at the item on a table.

[0237] Then, whether or not adjustment is necessary is determined based on whether or not the reflection state is appropriate. If it is determined that adjustment is necessary, the terminal device 1 proceeds from step S303 to step S304, and calculates what adjustments will be necessary to fit the subject appropriately in the second area image 25, and sets the movement mode. For example, the terminal device 1 sets the movement direction and amount, or the enlargement rate or reduction rate. In some cases, the terminal device 1 sets movement and enlargement, or movement and reduction.

[0238] At this time, it is desirable to set the detection target 26 to maintain a state in which it is within the second area frame 27. In particular, if the return frame 28 is not provided and the determination of whether the release condition is met determines that the detection target 26 has escaped from the second area frame 27, it is necessary to maintain the state in which the detection target 26 is within the second area frame 27.

[0239] In step S305, a process for changing the range of the second area frame 27 is performed in accordance with the settings. For example, the pixel range targeted for cropping is changed in accordance with the settings. Pan, tilt, and zoom control may be performed on the PTZ camera 2B. This allows the target subject to be properly captured in the second area image 25. In addition to adjusting the pixel range of the second area image 25, the second area frame 27 displayed on the whole image display unit 21 may also be moved, enlarged, or reduced. This is because the user 101 can recognize the adjusted second area frame 27.

[0240] The above is an example in which the second area frame 27 is automatically adjusted, but the user 101 may be allowed to adjust the second area frame 27 at will. A processing example is shown in Fig. 49. For example, the terminal device 1 performs the processing of Fig. 49 when the second area image 25 is set as the sent image P1S.

[0241] In step S320, the terminal device 1 monitors the movement operation of the second area frame 27. For example, it monitors whether the user 101 is dragging the second area frame 27 on the screen of the whole image display unit 21.

[0242] When a move operation is detected, in step S321, the terminal device 1 performs a move setting and necessary instructions for the second area frame 27. That is, the second area frame 27 is moved on the screen in response to a drag operation, and crop processing and pan / tilt / zoom operations are controlled so that the second area image 25 of the pixel range of the second area frame 27 at the move destination can be obtained.

[0243] In step S322, the terminal device 1 monitors the operation of enlarging or reducing the second area frame 27. For example, the terminal device 1 monitors the operation of pinching out or pinching in the second area frame 27 by the user 101 on the screen of the whole image display unit 21.

[0244] If an enlargement or reduction operation is detected, in step S323, the terminal device 1 sets and gives necessary instructions for enlarging or reducing the second area frame 27. That is, the second area frame 27 is enlarged or reduced on the screen in response to a pinch operation, and crop processing and pan-tilt-zoom operations are controlled so that the second area image 25 of the pixel range of the second area frame 27 after enlargement or reduction can be obtained.

[0245] This allows the user 101 to arbitrarily adjust the range of the second area image 25 even during shooting, and allows the second area image 25 to be sent in which the appropriate subject is appropriately emphasized.

[0246] 7. Summary and Modifications The above-described embodiment provides the following advantages. When the first area image 24 in a captured image is set as the send image P1S, the terminal device 1 of the embodiment performs send image change control to change the second area image 25 in the captured image to the send image P1S in response to detection that the detection target 26 associated with the second area frame 27 is included within the second area frame 27. In other words, the program of the embodiment causes the information processing device 70 to execute the above-described processing of the terminal device 1. This allows the user 101 to switch the send image P1S to be sent for distribution from the first area image 24 to the second area image 25 by causing the detection target set in association with the second area frame 27 to enter the second area frame 27. This allows the user 101 to control the switching to a cropped image, zoomed image, or the like as intended by the user 101 who creates the content to be distributed.

[0247] Furthermore, the second area indicated by the second area frame 27 functions as a detection frame for image switching operations and is also the range of a cropped image, a partially captured image, a zoomed image, or the like. In other words, the second area frame 27 is determined as the range of the transmitted image. Therefore, the user 101 simply sets the area they want to capture in close-up as the second area and moves the specific body part designated as the detection target 26 so that it enters that area. Specifically, if the right hand or right wrist is designated as the detection target 26 and the range in which the item 6 is captured when held in the right hand is designated as the second area, then when the user 101 holds the item in his or her right hand during a demonstration, the range of the item can be switched to a close-up image. In other words, switching to the second area image 25 is possible with a very natural movement. This improves the quality of the demonstration and achieves appropriate image switching.

[0248] As in step S252 in Figures 37, 38, and 39A, by determining that the detection target is within the second area frame 27 when the detection target remains within the frame for a predetermined period of time, frequent switching of the transmitted image P1S can be prevented when the detection target frequently enters and exits the second area frame 27 in a short period of time.

[0249] In the embodiment, the second area defined by the second area frame 27 is set as a portion of the first area. The terminal device 1 then displays the second area frame 27 indicating the second area within the image of the first area (within the entire image). For example, the second area frame 27 is displayed in the entire image display section 21 on the screen of the terminal device 1. This allows the user 101 to perform shooting for producing content to be distributed while recognizing the set second area. Note that the display form indicating the second area does not have to be a "frame." For example, the second area may have different brightness or color, or may be displayed as the four corners of a rectangular area.

[0250] Furthermore, although the second area is described as being inside the first area, the second area may only partially overlap the first area, or may be an area outside the first area.

[0251] In the embodiment, the detection target 26 is a specific part of the person being photographed. The terminal device 1 then executes control so that a display based on the skeleton detection of the person being photographed is performed within the first area image. For example, skeleton points 32 are displayed based on the skeleton detection results of the person being photographed. This allows the user 101 to clearly understand the position of the detection target in their own behavior. This allows them to take a photograph while appropriately understanding the position of the detection target (e.g., the left wrist) relative to the second area frame 27. This is effective for switching between the first area image 24 and the second area image 25 as the transmitted image as intended. Although the display based on the skeleton detection is a dot skeleton point 32, other display modes, such as a display showing skeletal lines, may also be used.

[0252] In the embodiment, an example has been given in which the terminal device 1 executes control of a cropping process to cut out the second area image 25 from the first area image 24 as the transmitted image change control. As the transmitted image change control, for example, the determination / instruction unit 50 of the terminal device 1 issues a crop instruction to the image processing unit 43 of the camera 2 (see FIG. 25) or issues a crop instruction to the image processing unit 57 of the terminal device 1 (see FIG. 27). This switches the transmitted image P1S from the entire image (first area image 24) to the cropped image (second area image 25). This allows the user 101 to arbitrarily switch the transmitted image P1S to be used as distribution content from the entire image (first area image 24) to the cropped image (second area image 25).

[0253] In the embodiment, an example has been given in which the terminal device 1 performs the transmitted image change control by instructing the PTZ camera 2B, which captures the image, to change the imaging direction and imaging angle of view from the first area image 24 to the imaging direction or imaging angle of view of the second area image 25 (see FIG. 29 ). This switches the transmitted image P1S from the whole image (first area image 24) to the zoomed image (second area image 25). The user 101 can arbitrarily switch the transmitted image P1S, which is to be the distribution content, from the whole image to the zoomed image.

[0254] In the embodiment, an example has been given in which the terminal device 1 performs the process of switching the transmitted image P1S from an image captured by the camera 2A1 capturing the first area to an image captured by the camera 2A2 capturing the second area, as the transmitted image change control. For example, as shown in the example of FIG. 31 , the determination / instruction unit 50 of the terminal device 1 instructs the switcher unit 59 to switch from the stream image P11 captured by the camera 2A1 capturing the first area to the stream image P12 captured by the camera 2A2 capturing the range of the second area frame 27. Alternatively, as shown in the example of FIG. 33 , the determination / instruction unit 50 instructs the switcher unit 59 to switch from the stream image P1A captured by the camera 2A capturing the first area to the stream image P1B captured by the PTZ camera 2B zooming in and capturing the range of the second area frame 27. This allows the user 101 to arbitrarily switch the transmitted image P1S, which is the content to be distributed, from the entire image (first area image 24) to a partial captured image or a zoomed image (second area image 25).

[0255] In the configuration example of FIG. 33 , the terminal device 1 performs control to change the transmitted image by switching to the image captured by the second camera after the PTZ camera 2B has completed the transition to the imaging direction and zoom angle of view for capturing the second area (see FIG. 46 ). When the PTZ camera 2B captures the second area image 25, if pan / tilt / zoom movement is performed, the switcher unit 59 is switched after the movement is completed. This enables switching without showing the zoom or pan / tilt process in the transmitted image P1S. In particular, multiple second area frames 27 can be set, and a single PTZ camera 2B can capture multiple second area frames 27. In this case, switching without showing the pan / tilt / zoom process is also possible.

[0256] In the embodiment, the terminal device 1 performs an example of control to change the sent image P1S by changing the sent image P1S from the second area image 25 to the first area image 24 in response to detection that the detection target 26 is no longer included in the second area. This allows the user 101 to change the sent image P1S from the second area image 25 to the first area image 24 by moving the detection target 26 outside the second area frame 27 (see FIGS. 14 and 40 ). Switching between the first area image 24 and the second area image 25 can be performed at will. By returning the sent image P1S to the first area image 24 when the detection target 26 continues to be outside the second area frame 27 for a predetermined period of time, as in step S262 of FIG. 40 , frequent switching of the sent image P1S can be prevented when the detection target frequently enters and leaves the second area frame 27 in a short period of time.

[0257] In the embodiment, an example was given in which the terminal device 1 executes control to change the sent image P1S by setting the second area image 25 as the sent image P1S and then changing the first area image 24 as the sent image P1S in response to detection that the detection target 26 is included in a third area set separately from the second area. For example, a return frame 28 may be set as the third area, and the user 101 may position the detection target 26 within the return frame 28 to return the sent image P1S from the second area image 25 to the first area image 24 (see FIGS. 15 and 41 ). This allows for arbitrary switching between the first area image 24 and the second area image 25. Furthermore, by setting the condition for switching to the first area image 24 as the detection target 26 entering the return frame 28 rather than exiting the second area frame 27, the user 101 does not need to worry about keeping the detection target within the second area frame 27 while the second area image 25 is being sent. Furthermore, as in step S264 of Figure 41, by returning the detection target to the first area image 24 when the detection target remains within the return frame 28 for a predetermined period of time, it is possible to prevent the sent image P1S from returning to the first area image 24 when the detection target suddenly enters the return frame 28.

[0258] In the embodiment, an example has been given in which the terminal device 1 performs control to change the sent image P1S by changing the second area image 25 to the sent image P1S in response to detection that the detection target 26 is included in a fourth area set separately from the second area when the first area image 24 is set as the sent image P1S. For example, a detection frame 29 is set as the fourth area, and the user 101 positions the detection target 26 in the detection frame 29, thereby switching the sent image P1S from the first area image 24 to the second area image 25 (see FIGS. 16 and 37). This allows the user 101 to arbitrarily switch from the first area image 24 to the second area image 25.

[0259] Note that if the detection frame 29 is set, switching to the second area image 25 may be performed either when the detection target 26 enters the second area frame 27 or when the detection target 26 enters the detection frame 29. Alternatively, if the detection frame 29 is set, switching to the second area image 25 may be performed when the detection target 26 enters the detection frame 29, without the entry of the detection target 26 into the second area frame 27 being a switching condition. In this manner, the user 101 does not need to pay too much attention to their behavior in the vicinity of the second area frame 27 while the first area image 24 is being sent. Note that in the case of the detection frame 29, frequent switching of the sent image P1S can also be prevented by determining that the condition is satisfied when the detection target remains within the return frame 28 for a predetermined period of time, as in step S252 of FIG.

[0260] In the embodiment, an example was given in which, when the terminal device 1 switches the sent image P1S from the first area image 24 to the second area image 25 by the sent image change control, an adjustment process is performed to adjust the position or size of the second area according to the subject. For example, even if the detection target 26 is placed in the second area frame 27, it is not clear whether the subject that the user 101 wants to show in close-up is included in the second area image 25. Therefore, by processing such as image recognition, the second area frame 27 is moved or enlarged / reduced so that the item that is the subject of the second area fits within the second area image 25 (see Figures 47 and 48). This makes it easy to send the second area image 25 with a close-up of the item or part that the user 101 wants to show.

[0261] In the embodiment, an example has been given in which, when the terminal device 1 sets the sent image P1S to the second area image 25, an adjustment process is executed to adjust the position or size of the second area in response to a user operation. While the second area image 25 is being sent, the user 101 can move, enlarge, or reduce the second area frame 27 (second area) by performing a predetermined operation such as dragging or pinching in / out of the second area frame 27 (FIG. 49). This allows the user 101 to adjust the second area image 25 to an image of an appropriate range depending on the current subject. Note that the type of operation is not limited to a drag operation or a pinch operation. Any type of operation may be used as long as it allows for an operation to instruct movement or enlargement / reduction.

[0262] In the embodiment, an example has been given in which the terminal device 1 displays some of the skeleton points 32 that indicate the parts of the subject person that have been set as detection targets within the image in the first area, as a display based on the skeleton detection of the subject person. For example, even if a large number of skeleton points 32 are detected as a result of the skeleton detection of the subject person, some of them, such as skeleton points 32 in or near the parts that have been set as detection targets 26, may be displayed (see FIGS. 21B and 35). This makes it possible to avoid the situation where too many skeleton points 32 are displayed, making it difficult to check them on the screen.

[0263] In the embodiment, an example was given in which the terminal device 1 executes a display based on the skeleton detection of the subject person based on the positional relationship between the body parts set for the detection target and the second area. For example, the skeleton point 32 is displayed when the detection target 26 of the subject person approaches the second area frame 27 or when the detection target 26 in the second area is about to move out of the second area frame 27 (see FIG. 36 ). This allows the user 101 to confirm the position of the detection target 26 when a switch between the first area image 24 and the second area image 25 is about to occur as the transmitted image P1S. This makes it easier to execute the switch as intended.

[0264] In the embodiment, an example has been given in which the terminal device 1 controls the display on / off based on the skeleton detection of the person being photographed in response to a user operation. For example, a skeleton display button 22 is provided so that the user 101 can arbitrarily turn on / off the display of the skeleton points 32. This allows the user 101 to select whether to display or hide the skeleton points 32 as needed.

[0265] In the embodiment, an example has been given in which the terminal device 1 displays, within the image of the first area, a display clearly indicating the correspondence between the display indicating the second area and the display indicating the detection target 26. A second area frame 27 indicating the second area is displayed, and the detection target 26 is indicated by a skeleton point 32. At this time, the correspondence is indicated, for example, by displaying the skeleton points 32 at the positions of the associated second area frame 27 and the detection target 26 in the same color. This makes it easier for the user 101 to understand which part of the body the detection target 26 is set to, and, when multiple pairs of second area frame 27 and detection target 26 are set, what kind of switching will occur when which part is moved to which position.

[0266] In the embodiment, an example was given in which the terminal device 1 causes the information processing device to execute control to simultaneously display on one screen a sent image P1S and an entire image of a captured image including a display indicating the second area. For example, as shown in Figures 12A and 12B, the terminal device 1 is provided with a sent image display unit 20 and an entire image display unit 21, which display the current sent image P1S (either the first area image 24 or the second area image 25) and the entire image as a captured image (e.g., the first area image 24) side by side. This allows the user 101 to simultaneously confirm whether the current sent image P1S is the first area image 24 or the second area image 25, and the positional relationship between the current detection target 26 and the second area frame 27.

[0267] In the embodiment, the terminal device 1 executes a setting process in which, in response to a user operation, the range of a second area within a first area is set and a detection target associated with the second area is set. For example, a UI using a setting screen 90 such as that shown in FIG. 6 allows the user 101 to set a second area (second area frame 27) of any position and size within the entire captured image (e.g., the entire first area image 24) (see FIGS. 7 and 8). This allows the user 101 to set the second area frame 27 that is convenient for their behavior during shooting.

[0268] In the embodiment, an example has been given in which the terminal device 1 executes a warning process when an unacceptable overlap occurs between the second areas when multiple second areas are set (see FIGS. 9 and 10 ). This prevents multiple second area frames 27 from being set in an inappropriate state.

[0269] The program of the embodiment causes the CPU 71 of the information processing device 70 to execute the control processing of the terminal device 1 of the embodiment described above. As a result, the terminal device 1 executes the processing described in the embodiment.

[0270] The program of the embodiment is a program that causes, for example, a CPU, a DSP (digital signal processor), an AI processor, or the like, or an information processing device 70 including these, to execute processes such as those shown in Figures 34 to 46, processes shown in Figures 48 and 49, and further processes such as those shown in Figures 7, 8, and 11.

[0271] With such a program, the information processing device 70 that functions as the terminal device 1 of the embodiment can be realized in, for example, a computer device, a mobile terminal device, or other equipment capable of executing information processing.

[0272] Such a program can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such a program can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0273] Furthermore, such a program is suitable for providing a wide range of information processing devices 70 that constitute the terminal device 1 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can be made into information processing devices 70 that function as the terminal device 1 of the present disclosure.

[0274] Although each process is executed by a program in the terminal device 1, all or part of the processes of the embodiment may be performed by an information processing device serving as a cloud server that can communicate with the terminal device 1. For example, all or part of the processes of the determination / instruction unit 50, the drawing unit 51, the rotation processing unit 53, the skeleton estimation unit 54, the frame setting unit 55, the rotation processing unit 56, the image processing unit 57, the image extraction unit 58, and the like, which have been described as functions of the terminal device 1, may be performed by the cloud server.

[0275] In addition, although the embodiment has been described as an example in which camera 2 and terminal device 1 are used, camera 2 may have the configuration of information processing device 70 as shown in Fig. 5, and may capture images while also executing the setting processing and transmission image change control described as processing of terminal device 1, thereby directly uploading video content to distribution server 3. In this case, monitor screen 15 during shooting may be displayed on a separate monitor device so that user 101 can view it.

[0276] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0277] The present technology may also be configured as follows: (1) A program causing an information processing device to execute, when an image of a first area in a captured image is set as a sent image, a sent image change control for changing an image of the second area in the captured image to the sent image in response to detection that a detection target associated with the second area is included in a second area set as a narrower angle of view range than the first area. (2) The program described in (1) above, causing an information processing device to execute control such that the second area is set as a part of the first area, and a display indicating the second area is performed within the image of the first area. (3) The program described in (1) or (2) above, causing an information processing device to execute control such that the detection target is a specific part of a person who is a subject, and a display based on skeletal detection of the person who is a subject is performed within the image of the first area. (4) The program described in any of (1) to (3) above, causing an information processing device to execute, as the sent image change control, control of a cropping process for cutting out an image of the second area from the captured image of the first area. (5) The program according to any one of (1) to (3), wherein the outgoing image change control is to cause an information processing device to execute control to instruct a camera that captures an image to change the imaging direction and imaging angle of view from the first area to the imaging direction or imaging angle of view of the second area. (6) The program according to any one of (1) to (3), wherein the outgoing image change control is to cause an information processing device to execute control to switch the outgoing image from an image captured by a first camera that captures the first area to an image captured by a second camera that captures the second area. (7) The program according to any one of (1) to (6), wherein the outgoing image change control is to cause an information processing device to execute control to change the image of the first area as the outgoing image in response to detection that the detection target is no longer included in the second area when the image of the second area is the outgoing image.(8) The program according to any one of (1) to (6) above, which causes an information processing device to execute control to change the image of the first area as the send image in response to detection that the detection target is included in a third area set separately from the second area when the image of the second area is set as the send image. (9) The program according to any one of (1) to (8) above, which causes an information processing device to execute control to change the image of the second area as the send image in response to detection that the detection target is included in a fourth area set separately from the second area when the image of the first area is set as the send image. (10) The program according to any one of (1) to (9) above, which causes an information processing device to execute adjustment processing to adjust the position or size of the second area in response to a subject when the send image is switched from an image of the first area to an image of the second area by the send image change control. (11) The program according to any one of (1) to (10) above, which causes an information processing device to execute adjustment processing to adjust the position or size of the second area in response to a user operation when the send image is set as the image of the second area. (12) The program described in (3) above, which causes an information processing device to execute control such that, as a display based on skeletal detection of a subject person, some skeletal points indicating parts of the subject person that have been set as detection targets are displayed within the image of the first area. (13) The program described in (3) or (12) above, which causes an information processing device to execute control such that a display based on skeletal detection of the subject person is displayed based on the positional relationship between the parts that have been set as detection targets and the second area. (14) The program described in any of (3), (12), and (13) above, which causes an information processing device to execute on / off control of a display based on skeletal detection of the subject person in accordance with a user operation. (15) The program described in any of (1) to (14) above, which causes an information processing device to execute control such that a display that clearly shows the correspondence between the display indicating the second area and the display indicating the detection targets is displayed within the image of the first area.(16) The program according to any one of (1) to (15) above, which causes an information processing device to execute control to simultaneously display on one screen a send image and an entire image of a captured image including an indication of the second area. (17) The program according to any one of (1) to (15) above, which causes an information processing device to execute processing to set the range of the second area within the first area and set a detection target to be associated with the second area in response to a user operation. (18) The program according to (17) above, which causes an information processing device to execute warning processing when an unacceptable overlap occurs between the second areas when a plurality of second areas are set. (19) An information processing device comprising: a control unit that performs send image change control to set the image of the second area in the captured image as the send image in response to detection that a second area set to have a narrower angle of view range than the first area includes a detection target associated with the second area, when the image of the first area in the captured image is set as the send image. (20) An information processing method in which, when an image of a first area in a captured image is set as the sent image, an information processing device performs sent image change control to set the image of the second area in the captured image as the sent image in response to detection that a detection target associated with a second area set as a narrower angle of view range than the first area is included in the second area.

[0278] REFERENCE SIGNS LIST 1 Terminal device 2, 2A Camera 2B PTZ camera 15 Monitor screen 20 Transmitted image display section 21 Whole image display section 22 Skeleton display button 23 Detection execution button 24 First area image 25 Second area image 26 Detection target 27 Second area frame 28 Return frame 29 Detection frame 32 Skeleton points 50 Judgment and instruction section 51 Drawing section 54 Skeleton estimation section 55 Frame setting section 57 Image processing section 58 Image extraction section 59 Switcher section 70 Information processing device 71 CPU 90 Setting screen 101 User

Claims

1. A program that causes an information processing device to execute a transmission image change control in which, when an image of a first area in a captured image is set as the transmission image, an image of the second area in the captured image is set as the transmission image in response to detection that a detection target associated with the second area is contained in the second area set as a narrower angle of view range than the first area.

2. The program according to claim 1, which causes an information processing device to execute control such that the second area is set as a portion of the first area, and a display indicating the second area is displayed within an image of the first area.

3. The program according to claim 1, which causes an information processing device to execute control so that the detection target is a specific part of the human subject, and a display based on detection of the human subject's skeleton is performed within the image of the first area.

4. The program according to claim 1, which causes an information processing device to execute control of a cropping process for cutting out an image of the second area from a captured image of the first area as the transmitted image change control.

5. The program according to claim 1, wherein the transmitted image change control causes an information processing device to execute control to instruct a camera capturing an image to change the imaging direction and imaging angle of view from the first area to the imaging direction or imaging angle of view of the second area.

6. The program according to claim 1, which causes an information processing device to execute control of a process for switching the transmitted image from an image captured by a first camera capturing the first area to an image captured by a second camera capturing the second area as the transmitted image change control.

7. The program according to claim 1, which causes an information processing device to execute control for changing the transmitted image, in which, when an image in the second area is set as the transmitted image, upon detection that the detection target is no longer included in the second area, the information processing device controls the image in the first area to be set as the transmitted image.

8. The program according to claim 1, which causes an information processing device to execute control for changing the transmitted image, in which when an image in the second area is set as the transmitted image, upon detection that the detection target is included in a third area set separately from the second area, the information processing device controls the image in the first area to be set as the transmitted image.

9. The program according to claim 1, which causes an information processing device to execute control for changing the transmitted image, in which when an image in the first area is set as the transmitted image, upon detection that the detection target is included in a fourth area set separately from the second area, the information processing device controls the image in the second area to be set as the transmitted image.

10. The program according to claim 1, which causes an information processing device to execute an adjustment process for adjusting the position or size of the second area according to the subject when the transmitted image is switched from an image of the first area to an image of the second area by the transmitted image change control.

11. The program according to claim 1, which causes an information processing device to execute an adjustment process for adjusting the position or size of the second area in response to a user operation when the transmitted image is an image of the second area.

12. The program described in claim 3, which causes an information processing device to execute control so that, as a display based on skeletal detection of the subject person, some skeletal points indicating parts of the subject person that have been set as detection targets are displayed within the image of the first area.

13. The program according to claim 3, which causes an information processing device to execute control so that display based on detection of the human skeleton as the subject is performed based on the positional relationship between the body parts set as the detection target and the second area.

14. The program according to claim 3, which causes an information processing device to execute on / off control of display based on detection of the skeleton of a person who is a subject in response to a user operation.

15. The program according to claim 1, which causes an information processing device to execute control so that a display showing the correspondence between the display indicating the second area and the display indicating the detection target is displayed within the image of the first area.

16. The program according to claim 1, which causes an information processing device to execute control for simultaneously displaying on one screen a transmitted image and an entire image of a captured image including a display showing the second area.

17. The program according to claim 1, which causes an information processing device to execute a process of setting the range of the second area within the first area in response to a user operation, and setting a detection target to be associated with the second area.

18. The program according to claim 17, which causes an information processing device to execute a warning process when an unacceptable overlap occurs between the second areas when a plurality of the second areas are set.

19. An information processing device having a control unit that performs send-image change control to set an image of a first area in a captured image as the send image when it is detected that a detection target associated with a second area, the second area being set as a narrower angle of view range than the first area, is included in the second area.

20. An information processing method in which, when an image of a first area in a captured image is set as the outgoing image, an information processing device performs outgoing image change control to set the image of the second area in the captured image as the outgoing image in response to detection that a detection target associated with a second area, the second area being set as a range of a narrower angle of view than the first area, is included in the second area.